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ABSTRACT 

Within a program of training system evaluation, 
optimal training system development cannot occur without evaluation 
development, and evaluation cannot be very effective in the absence 
of an optimal approach to training system development. Therefore, the 
goal of the Community Affairs Training Evaluation (CATE) project was 
to develop a set of procedures which provide practical information to 
decision makers for making program decisions and judging goal 
attainment. The result has been the development of a prototype for a 
Decision Oriented Evaluation System (DOES), involving evaluations of 
the training procedures and materials, the trainee's job performance, 
and the community impact of the program*- to be done in three stages: 
the input stage, the process stage, and the outcome stage, a 
prototype user's handbook has been developed and will allow the pilot 
testing revision, field testing, extension, and development of a 
supporting training system which are necessary before a comprehensive 
installation can be made. (Author/BW) 
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Introduction 

Background 

The development and execution of state-wide training programs 
require well informed decision-making in order to maximize effectiveness 
in reaching program goals. The Texas Department of Community Affairs 
(TDCA) has recognized this need for itself and has expressed the concern 
for decision-making capabilities based on sound program evaluation for 
national, state, and local levelij also. To assist TDCA in developing 
evaluation capabilities, the Community Affairs Training Evaluation (CATE) 
project was funded through an interagency agreement between TDCA and 
the University of Texas under a grant from the Department of Housing 
and Urban Development to be conducted by the Research and Development 
Center for Teacher Education at the University, The goal of the CATE 
project has been to provide TDCA with an evaluation system that can be 
used to judge the effectiveness of their training programs. More 
specifically, the project effort has been to perform the necessary 
research in the area of training evaluation and bring these findings to 
bear on developing a system that would assist TDCA to gauge the effective- 
ness of training, to examine the improvement of job performance as a 
function of training, and to investigate ways of determining impact of 
the community as a result of training.. 

Most people have fairly clear ideas of the concepts and operations 
of training. Much less clearer notions are held about evaluation. 
Indeed, the least clear understanding exists regarding the relationship 
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between training and evaluation and how closely these two operations are 
bound to each other. To get full measure from available resources, the 
trainer and evaluator must share perspectives. Optimal training system 
development cannot occur without evaluation development, and evaluation 
cannot be very effective in the absence of an optimal approach to 
training system development. Training development and evaluation cannot 
be divorced from each other without degrading both. What is needed 
then is a unified systems approach to both training and evaluation. An 
approach is needed that provides the context to understand the rela- 
tionships that exist between the parameters of training and those of 
evalua tion. 

The requirement of the system is to provide a unified set of 
procedures which will give decision-makers the information they need to 
determine answers to the following questions about their training pro- 
gram efforts : 

1. What kind of impact are training programs having on community 
deve lopment? 

2. Do the training programs improve the confidence and competence 
of the people they train? 

3. Are funding agencies getting adequate return for their invest- 
ment? 

The effort of this project has been directed toward providing 
decision-makers with information related to the above questions. The 
principle guiding the project has been to see evaluation as the process 
of identifying, obtaining, and using information to judge decision 
alternatives. From this definition of evaluation the GATE project's goal 



8 



3 



has been to develop a set of procedures which provide practical 
information to decision-makers for making program decisions and 
judging goal attainment. The result at the end of this contract year 
has been the development of a prototype for a Decision Oriented 
Evaluation System (DOES). The overall system design has been completed 
and elements of the system have been developed and pilot tested. The 
total system development is rather far from final completion and will 
require more pilot testing revision, field testing, extension and the 
developmenu of a supporting training system before comprehensive 
installation can be made. However, a prototype user's handbook has 
been developed and will allow a trial installation of DOES at TDCA in 
the summer of 1973 during which many of the field test que.'stions can 
be answered, and extension of the system can be made to TDCA ' s environ- 
ment. During this time development of a supporting system for installa- 
tion and training will be inaugurated with TDCA, 

DOES id entifies three mode s of evaluation. Each of these modes is 
concerned with providing evaluative information related to specific 
program efforts . 

Mode 1: Training . Evaluation in Mode 1 is applied to determine 
the quality of training and training materials per se , and is concerned 
with assessing skill development and the acquisition of confidence by 
trainees. Thus, evaluation is designed to determine whether the trainee 
can perform the tasks for which he has been prepared. 

Mode 2: Performan ce. The purpose of evaluation in this mode is 
to assess the effects of training upon the overall job performance of 
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the trainee once he has returned to his position. Here evaluation will 
provide information on specific application o£ skills developed during 
training, and whether the trainee has personally profited from his 
training. 

Mode 3: Community Impact . Mode 3 evaluation ±3 concerned with 
tracing the impact of training upon community development. Information 
supplied within this mode will relate to community benefits, systems 
effectiveness, responsiveness, service effectiveness and leadership 
deve lopment • 

An evaluation system focusing on these three modes used by 
decision-makers at the federal and state levels will assist program 
coordinators in: 



--setting guidelines for evaluation contracting 

— judging the quality of evaluation contracts 

--assessing the effectiveness of training programs 

--assessing the quality of training programs 

--tracing the effects of transfer of training to job per- 



formance and community development 
— providing guidelines for instructional training materials 
development • 

la addition, local decision-makers will use evaluation system 
outcomes to make decisions about programs and training appropriate for 
their communities. 

Decision-makers at all levels need procedures for translating 
evaluative information into a format which is practical for use by 
pe t:el untrained in evaluation. Therefore, the system will call for 
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evaluative informacion to be presented to the decision-maker in the 
following form: 

1. A one-page decision abstract which will include the following 
information: 

a. How good is the program (training activity)? 

b. What does it cost? 

c. how dependable are the data? 

d. What inferences can be mp.de? 

e. What recommendations can be made for action or decision? 

2, A four to six page abstract summarizing the technical report. 

3. A full technical report of the results of the study. 
General Approach 

In conceptualizing and designing the system, the project staff 
has drawn on several major resource areas: nodern educational evaluation 
theory and research, research in instructional systems design, training 
and evaluation development and applications in the military and private 
industry sectors, and industrial psychology. A massive amount of 
literature in these areas was gathered by the project's search and 
analysis team and analyzed by the staff. To obtain greater insight on 
developments and applications of training and evaluation methodologies, 
project teams visited numerous private industry, civil government, and 
military organizations. In addition, several GATE workshops (including 
two retreats) plus joint meetings of the GATE project and TDCA staffs 
were held to share concepts and findings and to modify developmental 
directions. Concepts and information gathered from the literature 
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survey and from site visits were analyzed and fed back on the original 
design configuration to evolve the system design shown in Figure 1, 

After the search and analysis and system design phases were well 
underway, development of specific evaluation instrumentation was begun. 
The major objective was to develop and pilot test instruments for the 
three evaluation modes of training, job performance, and community 
impact. Two training activities were selected as targets for pilot 
testing. These were supervision training and training for housing 
maintenance personnel. The supervisor training cc>urse was developed 
by the Engineering Extension Division at Texas A^.M and was used for 
p-^.lot testing training evaluation mode. The housing maintenance course 
was developed jointly by San Antonio College and the San Antonio Housing 
Authority and was selected for pilot testing instrumentation for job 
performance measurement and community impact observation. 

Ancillary to training evaluation was the development of a set of 
ratable criteria for the evaluation of training course materials 
constituting the training package being offered by the developer. Thus, 
the development of four different types of evaluation instruments was 
projected: training materials evaluation, training process evaluation, 
job performance measurement, and community impact observation. In 
addition to these evaluation instruments, the project undertook the 
development of two sets of criterial guidelines for sub-contracting: 
(1) the development of training activities and (2) the evaluation of 
training. 
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Rationale fot?S2S^c,^. • ^ ■ m ir i 

' ^s^din g ^^^^ Mode of Evaluation 

In order to answer *.r*.-^ 

Questions regarding the impact of training 

programs on community ^^^'^^{,^3^^, the extent to which training programs 
improve the competence and ^^^^.^^^^^ .^^ performance of trainees, 

and in order to provicJ^ f^^^>. ^ j i i • • i. • r 

^ ^al, state and local agencies with infor- 

mation regarding the return . . ^ ^ • ^ • . , 

^ their investment in training, we have 

proposed a three mode ^^^^"^ti^^ system. Evaluation is defined as the 
process of identifying, ^'''^^t^i^g ^nd using information to judge 
decision alternatives- 

The DOES evaluation f , ^ r i ^• 

^^m identifies three modes of evaluation. 

Mode 1 Is entitled Training ^. ^ . . , , ^, . , 

^ ^Valuation. Evaluation within this mode 

is concerned with the pref^^>j. _ • j ^ ^rr *.j ^ • • 

^Xsites for training, effective training 

procedures and with skill knowledge acquisition by trainees. 

Essentially, training e^al'^^ti^^ directed toward determining -hether 

or not trainees can peffo^^ bk ir '-u*.!.*.^^ i. u 

^ tasks for wnich the training has been 

designed. The second tnode O;^ . ^. . n^^ t u r, 4^ 

evaluation is called Job Performance 

Measurement. The purp^^^e k, . ^ 1 o*.- • 

t'^^F chis mode of evaluation is to assess 

the transfer effects of t^"^^^^* ^ u ^ ^ 

*<rtg to on-the-job performance or the 

trainees. The third mO<^e k , ^ ^ . ^ . 

termed Community Impact Evaluation. 

This mode of evaluation ^^tet^ ^, rc r 

*^ts to trace or observe the effects of 

training on community indic^^. rru • 4*. o*-*. *. *.u 

■'^ ^rs. That is, it attempts to answer the 

question "Are there anV ob^^^. ^ r.^ ^ , . 

M i-Mcri. any ^able community benefits which can be 

related in some way to train^^ ^^^^ training has improved the job 

performance of individuals w^^^^^ ^^^^^ government?'' 
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It should be inade c^^^r that community impact ^^^luatia^^ ^ ^^^V 
difficult field. First, is a new field of inquiry soci^'^ ^^i^-Hce 
research and little K^^°^n about it. a second ^^^^^^uity ^^ct 
that an enormous comple:^ °^ variables are dynamically -^t^tera^^^^^^ 
within a community; in n^^^t cases, attempts to isolate ^.^^ ^^feC^^ of 
specific training or Vajri^t>le« within that larger ^V^^^x^ co^te?^*^ 
be very difficult, if aC ^^l possible, f^om an evaluati^^ met^ho^J^^'^gy 
perspective. Indeed if ^°^unity impact evaluation ^^^ndt^^^^^ 
at all, it will prob^^^iy ^^ve to be done through in^^^e^^. ^^^u<i^^^ 
measures rather than ^it:^^^ measures. The most useful ^^^^ ^out:'^^^ 
are Ixkely to emerge ^ro^ Changes or trends in info^^t^Q^ eJ.ici*^^^ 
part of the natural proc^^^ record within a community b<j ],oc^^' 
state or national. 

The job pv:irform^nce ^Valuation system is a natural outgrr^wt:^ 
training evaluation. In ^^is regard, decision makers a^j^ irnP^^tant 
question "Did trainifiS a^^^Ually change the on-the-jo^ -forn^^"^^ 
individuals trained?*' J*^^ performance evaluation ^iff^^^^lt u^^^ss 
the training and the 3ob ^^clude specific observable t^e}^^^^^^s ^ork 
products that can be rat^^ against an external criterio,^^ ^^g^ <^^^n, 
the judgment of qualify t^^^omes a rather difficult issu^^ ^^^^ a 
methodological persp^c ti^^ ' 

Training evaluation P^r se is perhaps the easiest ^.^^-ee 
modes of evaluation. Ev^^ here, however, training ^^^^Uation 
function effectively ox\iy the extent to which ^^^^^ ^t^^cifi^^ g^^^^ 
have been established ^^^iningy specific objectives d^^^^ed 

from those goals, a^d sp^^^fic observable behaviors can gp^^^^^^^ 
as a function of training' i^Siess, these condition ^ £11 rne_t ^ 

O Id 
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exceedingly difficult to judge the effec^ of training^ and conduct 
training evaluation ^ In summary then the rationale for proceeding in 
three modes of evaluation is to ascertain whether training is producing 
the results intended, to determine if there is any influence on job 
performance as a function of training, and to determine if any changes 
in community life can be observed and in some way xciated to training. 
Stages of Evaluation Within Ka rh Mode 

In order to operationalize the three modes of evaluation it is 
necessary to add another dimension to the system. That dimension 
refers to the developmental stages of evaluation that must be opera- 
tionalized within each mode. The stages of evaluation are input _stage, 
process stage , outcome s_tajg£. 

The input stage refers to the gc' 'he ring of data and information 
prerequisite to the execution of both the process and outcome stages. 
The input stage is somewhat like the process of establishing readiness 
for a learning activity. Unless these input stage activities are con- 
ducted, within each mode, process and outcome evaluation stages cannot 
be conducted effectively and the entire evaluation process will fail, in 
all probability. An examination of the compon^^nts of the DOES system 
(see Handbook) will show that each mode of evaluation requires specific 
information for the input stage. 

The process stage of evaluation within each mode refers to those 
activities which take place while training evaluation, job performance 
measurement and community impact observation are in operation. The 
major objectives of the process stage of evaluation are: 1) to 
determine the observable consistency of training activities, job 
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performance observation and monitoring of community data with the 
stated objectives and goals; 2) to gather and feedback information to 
participants, developers and sponsors regarding the effectiveness of 
training, job performance and community observation; 3) to verify the 
training activities, job performance measurements and community impact 
indicators are congruent with the needs, objectives and outcomes 
specified in the input evaluation stage. 

The outcome evaluation stage refers to a set of procedures employed 
to gather information relative to the success of the effort within 
each mode. The outcome stage of evaluation is concerned then with 
the effectiveness of a training program in meeting its intermediate and 
its ultimate objectives, with the transfer of training to job performance 
and with the observation of imi^rovement of quality of service in a 
community as a function of training intervention. 

In summary, the input stage of evaluation provides the background 
for proceeding in each of the three modes of evaluation. The process 
stage of evaluation specifies those kinds of data to be gathered when 
the system is "in motion." The outcome stage of evaluation is concerned 
with gathering data relative to effectiveness. Outcome evaluation can 
be thought of in terms of payoff; that is, what returns are being 
realized for the investment in training: 1) in and of itself; 2) in 
terms of increasing the job performance of trainees; 3) with respect 
to producing observable changes in the community. 
Summary 

The three modes and three stages of evaluation provide the 
conceptual and operational framework for DOES- The modes of evaluation 
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represent the three areas of interest to the T^:<a.s Department of 
Community Affairs (TDCA) and the Housing and Urban Development (HUD). 
The stages of evaluation represent the activities that evaluators must 
undertake to provide the type of answers that TDCA and HUD seek. 
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Rationale for Training Package Evaluation and Criteria 
The evaluation of training is very closely related to the develop- 
ment and/or the evaluation of training materials. In most cases the 
quality of trainir'^ will bear a direct relationship to the quality 
of the training materials used. Therefore, the project staff developed 
criteria to rate training packages for potential use in community 
development training programs. These criteria would then be used to 
generate guidelines for subcontracting the development of training 
materials; the criteria would also be used to generate the evaluation 
model. The criteria are included in this report and in the Handbook, 
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Major Project Activities 
Of the many, varied tasks necessary to the design of a unified 
evaluation system, a few project activities warrant a brief recounting. 
The following activities produced important results for our system of 
evaluating training. 

(1) Initial Planning and Search Activities . For the first several 
months, the project's personnel were occupied principally with two tasks, 
project planning and search and analysis. With the assistance of all 
members of the staff, the planning specialist designed a Time Related 
Activity Chart (TRAC) which projected and integrated all tasks necessary 
to timely completion of the system. The search and analysis team 
collected and analyzr.d information related to training and evaluation 
of training at the managerial-supervisory level. At first members of 
the :eam interviewed directors of state, military and industrial training 
programs and collected training packages, manuals and materials. Their 
activities then focused on library resources that would widen the range 
of materials and permit specific examination of the actual processes 
of training, job performance and community impact. 

(2) The Sheraton-Crest Conference . The project team conceptualized 
the preliminary design of its evaluation system at a week-long conference 
at the Austin Sheraton Crest Inn. As outlined there, DOES provides three 
levels of information for evaluators and decision-makers: 

a. A system design for training evaluation to be used by 
professional evaluators. 
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b. Guidelines for developing training progt^^ms and for 
subcontracting evaluation; criteria for judgi^^g the 
quality of training development and evaluation. 

c. Decision memos for training evaluation In thr^^ modes. 

(3) The DOES Brochure . A mid-term progress report fu^^thered the 
conceptualization of DOES. At the request of the project's sponsor, 

the staff produced a ten page prospectus, which required a rnore thorough 
understanding of the evaluation system and its role In the training, 
evaluation and decision process. The resulting brochure ei^^ntAvely 
cotnmunicated the system's overall design to decision^-ni^ker ^ ^.'o. ".he- 
sponsoring agency. 

(4) Site Visits . Members of the staff suppleiuentr^d tn-iv -^sarch of 
printed sources with visits to training and evaluation specialists and 
decision-rnakers in the military, government ^nd industry- ^^t* example, 
research of the training mode was supplemented by vis:.^^i to South- 
western Bell, Texas A&M, San Antonio College and the s^n Antonio Housing 
Authority. Because the Air Force is expeciaHy effective at both 
training and job performance evaluation, project members visited the Air 
Training Command at Randolph Air Force Base ^nd both th^ Pe^^sonnel 
Research Division and the Occupational Measurement Squadron ^t Lackland 
Air Force Base. The limited printed sources on community i^^Pact were 
supplemented by visits to the Community University Research Associates 
and the Merrimack Education Center in Boston* to Urbar^ information 

and evaluations specialists in the City of hoS Angeles, the University of 
Southern California and the Rand Corporation* *to the highly effective 
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Service Bureau of the City of Dearborn, Michigan and to the Lyndon 
Baines Johnson School of Public Affairs at the University of Texas. 
(See Appendix D, "Site Visits Made By Project Team Members.") 

(5) The Lakeway Inn Conference . On March 5, 1973, the GATE staff 
met at the Lakeway Inn to formulate the project's Technical Report and 
Handbook and to assign the writing of the various sections of the 
project's final products. 

(6) Pilot Testing . 

C7) Meetings with the Office of Education, Information, and Training 
in the T exas Department of Community Affairs . To assure that system 
development was attuned to the sponsor's evaluation needs, the staffs 
of GATE and the Office of Education, Information, and Training met 
approximately once a month to review the progress of DOES and to exchange 
information and suggestions. Meeting topics included: project planning, 
TDCA's Title VIII training, future funding, recommendation's for OEIT 
training subcontracting, pilot testing and eval:jation workshops for 
the summer, 1973. 
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Search and Analysis 

The search and analysis division of the GATE project was created in 
order to provide information from a wide range of sources in an organized 
form. Its objectives were not only to prevent a "re-invention" of the 
wheel, but to learn from the ideas of others and offer information which 
would aid in the process of building something new. The search and analysis 
activities have focused on collecting and analyzing information related to 
training and to the evaluation of training at the managerial-supervisory 
level. During the fall of 1972, six interviewers were sent to state, 
military, and industrial settings in the Austin, Texas, area which were 
chosen on the basis of their development of training programs (see Appen- 
dix A), At this point the interviewers collected training packages, 
manuals and materials which could be used directly in a training program. 
It was decided that an examination of training and evaluation programs and 
materials in the immediately surrounding community would provide a realis- 
tic sampling of what we might expect to find in other information centers 
across the country. 

The training materials collected by the six interviewers were ini- 
tially screened on the basis of the following five criteria: (1) Are course 
expectations stated in behavioral terms? (2) Are there clearly specified 
roles for learners and instructors? (3) Has any evaluation of the program 
been undertaken? (4) Are instructional materials provided for learners? 
and (5) Has the training program been developed by a professional organi- 
zation or by qualified personnel? 

After this initial selection of training materials a detailed analysis 



22 



was made on the basis of the criteria listed below. These criteria were 
grouped into three divisions. The first division focused on Program 
Goals > Learning Experiences and Evaluation ; (1) Is there an explicitly 
stated general goal for the training package? (2) Are there specific objec- 
tives for each unit of the training package? (3) Is there a clear corres- 
pondence between the general goals and the unit objectives? (4) Are the unit 
objectives stated in learner behavioral outcomes? (5) Do the unit learning 
experiences appear to be consistent with the unit objectives? (6) Are the 
learners informed of the unit objectives which they are expected to acquire? 
(7) Are learners provided with an opportunity to practice the unit objec- 
tives they are expected to acquire? (8) Are provisions made for feedback to 
learners regarding their progress during the training? (9) Are provisions 
made for feedback to learners regarding their achievement at the end of 
training? (10) Are self-evaluation devices built in for learners to evaluate 
their progress en route. (11) Are performance evaluation measures available 
to determine learner achievement? and (12) Are performance evaluation measures 
congruent with unit objectives? 

The second division focused on the Characteristics of Instructional 
Materials and Activities : (1) Are the roles for instructors clearly de- 
fined? (2) Are the roles for learners clearly specif: \d? (3) Is training 
provided for instructors? (4) Are the training materials diversified? 
(5) Are the training activities diversified? (6) Are materials likely to be 
perceived as corresponding to the developmental level of the learners? 
(7) Are pre-tests provided for le^.mers? (8) Can the trainees start at an 
advanced level? (9) Can they finish at different levels? (10) Is the format 
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of the material conducive to easy use? (11) Is the presentation of material 
logically related to objectives? (12) Is the format of the material inter- 
esting to the reader? (13) Is the information included in the material 
accurate? (14) Is the information of importance to the training objectives 
(as opposed to being trivial)? and (15) Does the language usage lend clarity 
of expression? 

The third division focused on the Characteristics of Evaluation Mater- 
ials : (1) Are evaluation devices available? (2) Are evaluation items re- 
lated to learner performance objectives? (3) Are field test data available 
on evaluation instruments? (4) Are learners given the opportunity to practice 
on items similar to criterion items? (5) Are problem solving techniques used 
to measure learner behavior? (6) Have results of previous evaluation been 
built into the training package? and (7) Are self-evaluation devices built 
in for learners to evaluate their own progress? 

After analyzing the training materials collected by our interviewers of 
the state, military and industrial settings in the Austin area on the basis 
of the above criteria, it was concluded that the search and analysis acti- 
vities should be focused on library resources and materials that could be 
obtained through correspondence. Such resources would not only widen the 
range of materials, but enable a more specific examination of the processes 
involved in constructing a training and evaluation package oriented towards 
giving information to decision-makers. 

It was then decided that information searches should focus on the 
actual processes of training, job performance, and community impact, and 
continue the occupational focus on supervisors and housing managers. 
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The library reference sources and the major and minor descriptors 
used are listed in Appendix B. A bibliography is attached to this repoiL 
as well as an addendum to the bibliography for materials received up to 
the closing date for this stage of the GATE project, A variety of infor- 
mation sources were contacted by mail and a detailed listing of these 
sources appears in Appendix C. In addition to the information sources 
listed above, information was also retrieved through site visits made by 
project team members (see Appendix D) , 

The search and analysis team systematically retrieved information on 
topics related to the major and minor descriptors listed in Appendix B, 
Tnis material was analyzed not only from the standpoint of the criteria 
listed above with respect to training packages but also from the stand- 
point of its potential for aiding the GATE project team in constructing a 
decision oriented training evaluation package. This information was in 
turn categorized by major descriptors and subsequently presented to project 
team members for their examination in abstract form. Whenever abstracts 
were of particular interest to project team members, a hard ccjpy or micro- 
film copy was obtained either through correspondence or through information 
resources available in Austin, Texas, 

In order to make the materials readily accessible to project team 
members, all training packages, abstracts, hard copy and microfilm were 
categorized in the following general manner: 1) materials obtained 
through the University of Texas library system; 2) materials obtained 
through correspondence; 3) materials obtained through interviews; and 
4) materials obtained through TDCA or HUD, Within this general classifica- 



tion schema the materials were filed in the following categories: 1) 
training; 2) job performance; 3) community impact; 4) evaluation; and 
5) decision making. Materials that were judged to be especially relevant 
to our project goals were filed for immediate accessibility. Materials 
only indirectly related to project goals were labelled as "rejects" and 
filed separately. 

Another filing system was set up in order to keep a record of the 
location of all materials at any given time. A file oox was organized 
containing 3" x 5" notecards with references written in American Psycholo- 
gical Association style. The names of project team members requesting 
particular references were written on the back of the cards, along with 
the date at which the request was made. Separate reference cards were also 
kept for the vario; ' f ■ toembers to record their comments. This record- 
ing system made it ^ , xe for team members to remind themselves of refer- 
ences containing specit^ formation which interested them. 

The final part of the search and analysis record-keeping was a note- 
book containing the following information: 1) suggested procedures for 
search and analysis; 2) documentation of interviews; 3) correspondence; 
4) University of Texas library reference resources pertaining to the develop- 
ment of a training and evaluation package; and 5) search and analysis memo- 
randa and information circulated among the GATE project team. 
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PART II 

Training Evaluation 

Job Performance Evaluation 

Commurity Impact Observation 
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TRAINING EVALUATION 
Introduction 

The purpose of this section Is to show the relationship between training 
and evaluation as they fit into The Decision Oriented Evaluation System; and 
to illustrate this relationship by an application of evaluation to a train- 
ing activity. The section begins with an overview of training and evaluation. 
This overview is followed by a more precise statement of training as imple- 
mented by the Texas Department of Community Affairs. Finally, the general 
approach to training evaluation formulated by the GATE project is 
presented. 

Training — The State of the Art (General) 

Background 

Training can be broadly defined as the passing of knowledge and skills 
to people who don't have them. Earliest forms of training probably Involved 
primitive men wao had mastered their circumstances showing others around 
them how to use simple tools and weapons. Stelnmetz (1967) traces the evo- 
lution of training from about 3500 B.C. to 1966. He points out that up to 
the Industrial Revolution in 1790, training was primarily that of skill 
development of craftsmen. The apprenticeship system and the formation of 
guilds were critical developments in organizing the training of craftsmen. 

Major iirpetus in training development was provided by the need to ob- 
tain skilled workers to support World War I manufacturing effort. To meet 
the need for 600,000 shipyard workers, Charles Allen, the head of The Emer- 
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gency Fleet Corporation of the United States Shipping Board, implemented 
his four-step method of job instruction training — (1) Show, (2) Tell, 
(3) Do, and (4) Check. Drawing on the experience gr-ined in training workers 
in World War I, industry in the United States refined and applied the same 
strategy to meet the World War II crisis. This effort resulted in the per- 
manent establishment of training departments in industry. 

Coupled with the increasing importance of training in industry was 
the recognition by the i-ederal Government that training was vital to the 
vjell-being and survival of the United States. This importance is exem- 
plified by the Smith-Hughes Act of 1917 which provides $7 million annually 
for vocational education and more recently (1962) by the Manpower Develop- 
ment and Training Act focused on training the unemployed (Steinmetz, 1967). 

The critical role of training in the military has been underscored 
by Clark and Sloan (1964) who estimated that there are more people in train- 
ing in the armed forces than the combined enrollment of all universities 
and colleges in the United States. In fact, the United States Air Force 
alone devotes over $1 billion to training each year (Air Training Command 
Fact Sheet, 1972). 

At present (1973), the state of the art of training can be analyzed by 
examining training research and training development. 
Research 

Although the research reported in Psychological Principles in System 
Development (Gagne, 1962) and The Engineering of Educational and Training 
Systems (Smith, 1971) addresses socie of the methodological and systems 
problems involved in training, operationalization of their techniques re- 
mains limited and has not yet been comprehensively applied and evaluated. 
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The importance of training in the United States appears to be perva- 
sive, and yet research focused on training seems meager by comparison. For 
example, Wenig and Wolansky (1972) report in their review of the literature 
on job training in industry that: 

(1) Studies to fill the information gap on the scope and 

nature of job training in industry need to be made, 
(5) Since most workers are informally trained on the job 
and since very little, if any, information is avail- 
able about this type of job training in industry, 
it appears that research is needed about the rela- 
tionship between informal and formal training upon 
selected types of workers in various settings and 
occupational groupings to determine which type, 
either informal or formal training, is more effec- 
tive for a particular type of worker or in parti- 
cular industrial or business settings, 
(7) Research is needed for developing more imaginative 
planning and programming in the area of upgrading. 
Patterns are necessary for breaking down skill levels 
to a simpler category, (pp, 47-48) 
Development 

A major advance in the development of training has been made by the 
military and is exemplified by Smith (1971), He advanced an instructional 
system which is based on the systematic identification and specification of 
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tasks necessary in the performance of a given job. Specific performance 
objectives are derived from task analyses and subsequent training activities 
are developed to maximize the probability that a student will achieve the 
stated objectives. An evaluation component which constantly provides trainers 
with information regarding the degree to which objectives are being success- 
fully achieved is central to the training system. A low success rate gener- 
ally indicates that something is wrong with the instruction and not the 
s tu den ts . 

In spite of the fact that the literature remains replete with unanswered 
questions about training, significant trends in improving training have been 
found (Wenig and Wolansky, 1972). They cite the following trends reported 
by Lippitt (1969): 

1) Effectiveness of training based on worker performai. 
and not merely increased trainee knowledge. 

2) Industry developing in-house job training capabilities 
rather than depending on outside experts or resources. 

3) More homogeneity in training groups so that trainees 
learn to function together as members of a cooperative 
organization. 

4) Instead of vague assurances that training will be 
desirable for trainees, greater emphasis on goal 
orientation. 

5) Learning that is trainee self-activated rathei than 
imposed on him. 

6) The application of reinforcement strategies and follow- 
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up experiences for trainees which provide more lasting 
learning. 

7) Reduction in the use of structured nonparticipat ive 
approaches to learning and more activities that are 
action oriented, 

8) Planning and developing training activities that focus 
on learning how to learn, 

9) Demands of evaluation of training instead of accepting 
the results on faith. 

Evaluation 

Evaluation of training has taken a number of different forms. The 
methods used to evaluate training have ranged from asking a person how he 
liked the training to precise observations of a trainee ^s execution of the 
complex skills necessary to perform a job task. The first method has been 
criticized because it provides little or no information regarding the 
"effectiveness** of the training; and the second method of evaluation men- 
tioned, although apparently quite valid, is very expensive. The mid-ground 
used to obviate these criticisms has been the use of some type of objective 
test to collect information which is thought to approximate actual performance 
requirements • 

The shift from norm- referenced tests (like those used in most colleges 
and secondary schools) to criterion-referenced tests has been evident in job 
training. It has become blatantly apparent that if the objective of training 
is to equip a worker with the skills necessary to operate a drill press, 
comparing his score on some test with scores made by other trainees would 
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provide precious little information about how well the worker can operate a 
drill press. Use of performance criteria, although not a new concept, is 
beginning to enjoy widespread support from the developers of training pro- 
grams. (This is especially true in the military.) Consequently, the evalua- 
tion of a training program or course in which performance objectives have 
been specified has become somewhat less complex. There are, however, 
notable problems remaining for those involved in evaluating training. 

One problem deals with the validity of "test" situations compared to 
using actual job situations in determining the effectiveness of training. 
The further removed a trainee is from the real task, the more difficult it 
becomes to trace the transfer of training. 

A second problem focuses on how well the trainee performs his job tasks 
after the completion of training. There are many possible intervening fac- 
tors which may obfuscate the actual effectiveness of training, such as the 
trainee's attitude or interpersonal problems with peers or supervisors. 
Other facts which may add to the complexity of adequate evaluation are 
related to the levels of training performed. The more skill oriented the 
training is, the more likely adequate and informative evaluation is. Thus, 
the design of effective training must take such factors into -consideration 
and evaluation must account for these factors. 

Training and Evaluation (General) 

Training Package 

A critical assumption underlying the development of training packages 
is that training is necessary to change job performance. Thus, some imme- 
diate questions are raised: (1) What kind of training is necessary; (2) 
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At what level should training be focused; (3) What level of training should 
be applied; (4) What are the general goals of the training; (5) What are the 
specific objectives of training; (6) What precise behavior or performance 
do we want changed; and (7) Exactly how do we want behavior or performance 
to look at the end of training. If answers to these questions are provided, 
the design and subsequent evaluation of training packages is straightforward. 

Goals of training . There are a number of specific procedures which 
may be used to determine the goals of training. These procedures fall 
under the general term of "needs assessment". Any training goal, therefore, 
should be derivable from and directly related to problems identified by 
needs assessment. In this case, a problem may be defined as the discrep- 
ancy between the way things should be (what is desired) and the way things 
actually are. If specific discrepancies cannot be identified, the design 
of training may be premature and an inefficient use of resources will 
result. If discrepancies can be specified, the goals of training should be 
directly related to the discrepancies and internally consistent with them. 
Thus, goals of training should be derivable from specific problems. 

Objectives of training . The objectives of training should be derived 
directly from the goals of training in the same manner that goals of 
training should be directly related to specific problems. Whereas the goals 
of training are stated in general terms, the objectives of training should 
be stated in much more precise language. Preciseness may be obtained by 
stating desired training outcomes in terms of behavioral objectives. A 
behavioral objective usually specifies the precise behavior to be performed 
(and observed), the conditions under which the performance will take place, 
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and the crleerl r to be used in Judging the adequacy of the performance. 
Levels of Training 

The identification of training goals and objectives will generally pre- 
scribe the levels of training necessary to enable trainees to obtain the 
desired training outcomes. There are three major levels or outcomes of 
training. At the lowest level, outcomes may be specified in terms of "making 
trainees aware of", or "making trainees familiar with" some aspect of their 
job. The next level of training is focused on having trainees acquire and 
have knowledge about job related information. The highest level of training 
is aimed at providing specific skills in trainees to enable them to perform 
their job tasks more effectively. Once the levels of training have been 
determined for obtaining specific objectives, specification may be made of 
the levels of evaluation appropriate for judging training success. 
Levels of Evaluation 

The evaluation of training must begin with a consistency analysis be- 
tween training goals and training objectives. A consistency analysis sys- 
tematically evaluates the degree to which needs, goals, objectives, and 
training derive from one another and are congruent with each other. If 
there is little consistency among the needs, goals, objectives and train- 
ing, further evaluation is inappropriate. 

Once consistency between goals and objectives has been verified, 
specification of evaluation levels may begin. There are three major levels 
of evaluation. The purpose of evaluation at the first level is to deter- 
mine if trainees have acquired and comprehended information given them. 
At the second level of evaluation, trainees are asked to interpret and 
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apply the information they have received. The third level of evaluation 
calls for the trainees to demonstrate how adequately they have mastered the 
skills they have been taught. The degree to which training objectives are 
achieved is indicative of the success or adequacy of training. Evaluating 
training success is, therefore, directly related to the specific objectives 
of training. 

Equivalence of Levels of Training and Evaluatic ' 

Although it is possible to use different levels of evaluation for 
any level of training, the effectiveness of training may most adequately 
be determined by designing evaluation for the level at which training is 
provided. Once again, however, the training objectives and the training 
activities should be internally consistent. If the purpose of training 
is to familiarize trainees with a certain set of procedures which are to 
be used on their jobSj, it would be inappropriate to measure the degree to 
which trainees could reproduce the content of those procedures. On the 
other hand, if the purpose of the course is for trainees to use a set of 
procedures in performing a certain task observing their performance in 
following and using these procedures would be appropriate evaluation. 

Since the general goal of training is to "close the gap" between how 
things should be and the way things actually are, it is most desirable 
to use evaluation to provide evidence that this gap is indeed being 
closed. If this goal is to be achieved, care should be given in applying 
levels of evaluation which are consonant with training levels. 
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GATE General Approach to Training Evaluation 
The major underlying assumption that Is made with respect to training 
evaluation Is that the training activity will result In behavioral changes. 
It Is further assumed that these behavioral changes can be traced to more 
adequate job performance and that more adequate job performance will enable 
an organization to more adequately achieve Its goals. These assumptions 
are viable only when the following conditions are met: 

1) The goals of the organization are achievable and are 
stated In measurable terms; 

2) The training goals are derivable from organizational 
goals and are also directly related to job performance; 

3) The training objectives are derivable from training 
goals and are stated In job performance outcome be- 
havior based or job task analyses; and 

4) Training Is focused on Increasing the competencies 

of trainees to perform skills which are necessary for 

adequate job performance. 
If the above assumptions are valid and If the necessary conditions are 
met, adequate evaluation Is possible. Thus, the degree to which evaluation 
is effective Is directly related to how many assuiiq)tlons and conditions of 
training are met. The fewer the number of assunq)tlons and conditions met, 
the more limited the adequacy of the evaluation. 
Goals of Training 

Among the first steps in evaluating training is to determine the ade- 
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quacy of training goals. Some of the major questions which are asked in this 
phase of training evaluation include: 

1) Are the organizational goals stated in measurable terms - 
can accomplishment of these goals be determined? 

2) Is there a discrepancy between the goals of an organiza- 
tion and what is actually being done? 

3) Can the discrepancy be resolved by training, or is there 
a more efficient and effective way of closing the gap 
between the w ^ things are and the desired goals? and 

4) Are training goals directly related to organizational 
goals? 

Adequate answers to these questions are prerequisite to any further 
evaluation being effective. If the analysis of answers to these questions 
determines that the problem is one of training and that the goals of training 
are specified in observable terms, the evaluation of training objectives 
may be performed. 
Objectives of Training 

The cornerstone of evaluating the appropriateness of training objectives 
is the degree to which these objectives are derived from training goals and 
the degree to which these objectives are based on task analysis data and 
job performance requirements. Central to evaluating this phase of training 
are the following questions: 

1) Are the objectives of training directly related to the 
goals of training? 



38 



32 



2) Are the objectives of training based on task analysis data 
and job performance requirements (is there a discrepancy 
between job performance requirements and task analysiQ 
data)? 

3) Are training objectives stated in behaviorally observable 
terms? and 

4) Given time and money constraints, can training activities 
be designed which have a reasonable probability of enabling 
trainees to achieve these objectives? 

Satisfactory answers to the foregoing questions will generally provide 
the guidelines and standards necessary for the development of training ac- 
tivities and subsequent training activity evaluation. 
Training Materials and Activity 

Background infomation » To make effective decisions about training 
and training materials, the decision- maker should know something about the 
basic elements of training systems development. This basic information will 
help him in judging the quality of training materials and in developing 
guidelines for the subcontracting of training materials, 

A training system is conceived of as an integrated set of resources, 
human and material, designed to achieve specified outcomes in learners. 
The outcomes of any training program should be a set of skilled behaviors 
related to the job performance requirements within an occupation. The 
development of a training program must start with a thorough analysis of 
the tasks required to perform a job, and proceed through a sequence of 
specific steps that terminate with verification of the training system as 
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producing the results intended in trainees. The following model used by 
Air Training Command (1970) is illustrative of the process. 



Insert Figure labout here 

A brief description of the Air Force model components follows: 

Analyze System Requirements ; This step requires the developer to 
identify job tasks within the larger context of the system within which the 
job is to be performed. The analysis must describe the setting within 
which jobs are to be performed and all the interpersonal and/or man-machine 
relationships related to the environment of the operational system. 

2. Define Education or Trainin:^, Requirements : This step follows an 
analysis of job performance requirements and identifies the duties and 
tasks requiring instruction. The level of difficulty of tasks, resources 
available, and development time required are identified during this phase. 

3. Develop Objectives and Tests : Criterion behaviors desired as 
end products of training are identified and enabling objectives (prerequi- 
site knowledge, skills and attitudes) are specified. Performance tests are 
developed to reflect the job behaviors that are the object of training. 

4. Plan, Develop, and Validate Instruction : The sequencing of instruc- 
tion, the selection of instructional materials and methods, the development 
and testing of program prototypes comprise this design phase of the cycle. 

5. Conduct and Evaluate Instruction : The field testing of an in- 
structional system with particular emphasis on data based revision* or re- 
design of the instructional sequence and materials is a critical step. The 
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developer must rework the system until it produces the intended results with 
learners . 

The most critical element, then, in a training program is the clear 
specification of job requirements, training objectives related to require- 
ments, instruction designed to produce the behavioral outcomes, and evidence 
that the behaviors have been produced. 

Smith's (1966) model of an instructional system brings one closer to 
the process of conducting a training program and includes some important 
principles of training. A presentation of the model is displayed in Figure 3 
and a brief discussion of its elements follow: 



Insert Figure 2 about here 

From an an;=i lysis of the job performance requirements of trainees accord- 
ing to Smith's design, the developer selects specific behavioral objectives 
for the training program. Practice of performance refers to that aspect of 
training which engages the learner in the practice of the tasks and the 
skill components of training in a job simulation setting. Practice of 
knowledge provides for the learner to use the symbolic processes (e.g., key 
words, terms, concepts, diagrams) related to job performance and to receive 
reedback or knowledge of results throughout training. Presentation of 
knowledge refers to the means by which knowledge is transmitted to learners. 
Lectures, reading, graphic aids, and films are among tb^ most commonly used 
presentation modes. Management of students refers to the techniques used to 
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Figure 2 

A Model of an Instructional System 

(Smith, 1966, p. 7) 
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keep students participating productively in the learning experiences. Quality 
control refers to the processes employed to gather data on the effective- 
ness of the training system in accomplishing its objectives. An equally 
important aspect of quality control is to provide guidance regarding how the 
instructional system might be changed to increase its effectiveness. 

One other important ingredient of a training system is the specification 
of standards of proficiency that trainees should attain at the end of course 
instruction. These standards may also reflect a developmental process by 
which a trainee progresses through levels of job performance (i.e., appren- 
tice, journeyman, master). Proficiency levels at end of training depend 
upon the entry characteristics of thj trainees and may be perceived as part 
of a longitudinal effort that will be combined with on-the-job training 
to ultimately reach desired levels of job performance. 

Background information summary . The purpose of this section has been 
to provide the decision-maker with an orientation that will help him see the 
relationship of an isolated or short term training program, to the overall 
process of the design of an instructional system that is based on an 
analysis of job performance. These concepts are presumed to be helpful in 
assisting the decision-maker to judge the quality of training programs and 
prepare specifications for the development of training materials. 
Training Materials Evaluation 

Background information . The guidelines that follow will provide the 
decision-maker with key ideas related to the development of training mater- 
ials, or to the evaluation of training packages. These guidelines have been 
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derived from an extensive analysis of the literature on training materials 
development, and from the reported research. It needs to be r -ned 
that there are few empirical based generalizations that can s te^ about 
the effectiveness of specific classes of media related to tr^. ning astruc- 
tional effectiveness: this is largely due to methodological problems in 
instructional research. For example Baker (1973) reports that there is 
little evidence to support the notion that given a media as media provides 
for more effective learning. Briggs (1967) does point out, however, that 
multi-media instruction has proved to be superior to single-media instruc- 
tion. 

In the absence of generalizable, empirical based guidelines to support 
training materials development, one must turn to the examination of specific 
training materials that have undergone evaluation and do produce the desired 
results in learners. Examination of these ^materials provides a basis for 
development and also provide guidelines criteria that could be applied to 
the analysis of training materials. 

In summarizing the available literature on training and in drawing 
extensively on the training research experience of HumRRO, Smith (1966) 
states some guidelines related to this instructional system design: 

The critical aspects cf practice of performance are: 

a. to simulate the job task, using a detailed description as a 
guide 

b. to provide for knowledge of results 

c. to arrange a suitable practice schedule 

d. to oiaximize the transfer of training. 
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The critical aspects of practice of lenowledRe are: 

a. to determine, through analysis, the relation between cues 
and responses required by the knowledge 

b. to develop, through practice with knowledge of results, 
a high level of achievement 

c. to devise ways of making material meaningful to trainees. 
Presentation of knowledge can be done successfully by any of several 

methods provided: 

a. the presentation communicates to the student 

b. the material presented is meaningful 

c. the special characteristics of media are taken iuLo account. 
Instructional devices and media should be selected in terms of cost 

and effectiveness.. 

The Air Force Manual on Instructional System Development (AFM 50-2) p 
vides an excellent overview of the principles and practices that one ought 
to take into account when planning, developing and validating instruction. 
In presenting options to be pursued in the selection of media, the manual 
presents a very useful adaptation of a media selection matrix developed by 
Allen (AFM 50-2, pp. 5-14). 



Insert Table 1 about here 

The ratings of high, medium and low in Table 2 refer to the degree of 
suitability of the instructional media when conrpared to the objectives of 
learning . 
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TABLE 1 

Instructional Media Stimulus Relationships to Learning Objectives 





LEARNING OBJECTIVES: 


iirE Or 

INSTRUCTIONAL 

JIEDIA 


Learning 

Factual 

Information 


Learning 
Visual 
Identifi- 
cations 


Learning 
Principles, 
Concepts, 
and Rules 


Learning 
Procedures 


Performing 
Skilled 
Perceptual- 
Motor Acts 


Developing 
Desirable 
Attitudes, 
Opinions, < 
Motivation; 


Still Pictures 


Medium 


HIGH 


Medium 


Medium 


low 


low 


Motion Pictures 


Medium 


HIGH 


HIGH 


HIGH 


Medium 


Medium 


Television 


Medium 


Medium 


HIGH 


Medium 


low 


Medium 


Training Aids 


low 


HIGH 


Medium 


Medium 


low 


low 




Medium 


low 


low 


Medium 


low 


Mediun 


Drainer 
[Simulator) 


Ledium 


HIGH 


HIGH 


HIGH 


HIGH 


Medium 


Programmed 
Instruction 


Medium 


Medium 


Medium 


HIGH 


low 


Medium 


)emonstration 


low 


Medi\im 


low 


Medium 


Medium 


Medium 


Printed Textbooks 


Medium 


low 


Medium 


Medium 


low 


Medium 


)ral Presentation 


Medium 


low 


Medium 


Medium 


low 


Medium 



bte. — This table is adapted from Dr. Allen's paper, "Research in Instructional Media and Art 
Education," which was originally published in August 1966 in Final Report of the Uses 
of Newer Media in Art Education Project by the National Art Education Association, NEA, 
Washington, D. C. 
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Given this brief overview of characteristics of training systems, the 
criteria displayed in Table 2 and Table 3 will prove to be helpful in 
analyzing the potential effectiveness of training materials and/or training 
programs. Table 3 is adapted from Smith (1966, pp. 83-85). 



Insert Table 2 and Table 3 about here 



The checklists displayed in Table 2 and Table 3 provide the decision- 
maker with important characteristics of training programs/materials that 
should be taken into account. It might be helpful to present these charac- 
teristics in the form of Good, Better, Best for classification purposes. 
Characteristics of Training Programs 

Good. A good training program will have: (8-24 hours of instruction) 

1. A statement of outcomes in learner behavioral terms that: 
. lend themselves to observation or measurement 

. have a direct bearing to job perf orjmance 
. are consistent with the goals of training and with the 
training strategies. 

2. A variety of instructional materials and/or techniques 
which actively involve the trainee. 

Better. A better training program (24 to 40 hours of instruction) in 
addition to 1 and 2 above will have: 

3. Provisions for trainees to practice (simulate or role 
play) the behaviors for which they are being trained. 

4. Explicit provisions for self-evaluation and feedback 
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their progress during the training? 

8. Are provisions made for feedback to learners regarding 
their achievement at the end of training? 

9. Are self -evaluation devices built in for learners to 
evaluate their progress en route? 

10. Are performance evaluation measures available to deter- 
mine learner achievement? 

11. Are performance evaluation measures congruent with unit 
objectives? 

Characteristics of Instructional Materials and Activities 

1. Are the roles for instructors clearly specified? 

2. Are the roles for learners clearly specified? 

3. Is training provided for instructors? 

4. Are the training materials diversified vith respect to 
media and instructional methods? 

5. Are the training activities diversified for learners? 

49 



6. Are materials likely to be perceived as corresponding 
to the developmental or maturity level of the learners 

/. Arr prt-Lcsi:H prc^vided for learners? 

8. Can the trainees start at an advanced level? 

9. Can they finish at different levels? 
Characteristics of Evaluation Materials 

1. Are evaluation items and/or devices available? 

2. Are evaluation items related to learner performance 
obj ect ives? 

3. Are learners given the opportunity to practice during 
training on items similar to criterion items? 

4. Are problem solving techniques used to measure 
learner behavior? 
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TABLE 3 

A CHECK LIST FOR EVALUATING TRAINING 
(Correct Answers Are Capitalized) 

1. Obtaining information concerning the job for which the 
student is being trained. 

a. Is there a procedure for obtaining information 

about the job? YES no 

b. Is the procedure applied systematically and 

consistently? YES no 

c. Does the procedure collect performance information 

for meaningful units of activity? YES no 

d. Is performance information actively sought from 

sources in the work or life performance situation? YES no 

e. Is performance information recorded? yes NO 

f . Is performance information used systematically 

and consistently to identify critical instructional 

needs? YES no 

g. Does the procedure provide complete coverage of all 
likely aspects or occurrences of the desired work or 

life performance situation? YES no 

h. Does the procedure identify performance actions, con- 
dition, and standards relevant to the work or life 

situation? YES no 

2. Identifying specific training objectives. 

a. Are decisions about what to teach made on the basis 

of reliable and valid data? YES no 

b. Are detailed analyses made of tasks to be taught as 

a basis for identifying knowledges and skills required 

for task performance? YES no 

c. Are all skills and knowledges required for task per- 
formance identified? YES no 

d. Do training objectives state precisely the performance 

actions, conditions and standards? YES no 

e. Do specific training objectives use vague terms j such 
as know, understand, appreciate, familiarize, general 

knowledge, working knowledge, qualified? YES no 

3„ Establishing the sequence of instruction. 

a. Is there an effective orientation of the student to the 

entire job to be learned? YES no 

b. Are there blocks of skills and knowledge taught in iso- 
lation from their use in job tasks? yes NO 

c. Are new skills and knowledges taught only when required 

in order to master a new task? YES no 

d. Is the learning of new knowledge followed immediately 

by practical exercises? YES no 

e. Is the relation of each new task to be ..aarned to the over- 
all job clearly stated to the student? YES no 
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. Designing situations for the practice of performance. 

a. Are practice situations based on an analysis of 
the task to be learned? 

b. Does the student practice the entire task? 

c. Has any part of the task been omitted from practice 
Do training rl(r''.<.QH simulate the task? 

e. Dn inHlrucil^ nij for effective use accompany the 
training device? 

f . Has the chaining device been evaluated in terms 
of de\ -lojlv.g student proficiency? 

g. Have ci-aining devices vs real equipment been 
subjected to cost-effectiveness analysis? 

h. HciS the possibility of using obsolete equipment 
to teach appropriate skills been considered? 

i. Do trainees receive frequent and immediate knowledge 
of the effectiveness of their practice? 

j. Do trainees receive at least one minute rest 

between practice trials? 
Designing situations for the practice of knowledge. 

a. Is the knowledge to be practiced clearly related 
to an actual job task? 

b. Has information representing the job cues pro- 
vided the student, and the responses he Is to 
make, been identified? 

c. Has a practice session been planned for? 

d. Have appropriate practice materials (workbooks, 
self-instructional programs, flash cards, etc.) 
been designed? 

e. Do trainees receive frequent and immediate 
knowledge of the effectiveness of their practice? 

f. Do trainees maintain a record of their progress 
during practice? 

Preparing presentations to the student. 

a. Has the content of the presentation been tested 
on students to determine, by means of achievement 
tests, whether it communicates to the students? 

b. Is the content of the presentation meaningful 
to the student? 

c. Are there lengthy periods of presentation unin- 
terrupted by practice? 

d. Are films ard television integrated with live 
instruction? 

e. Are lectures, demonstrations, films, television 
or tape recordings selected on a cost-effective- 
ness basis? 

f. Have texts been examined to be sure that they 
are within the reading capability of the student? 

Maintaining student le^ rning activity. 
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a. Has the degree of spread in aptitude scores of the 

trainees been determined? YES no 

b. Have adjustments been made to the training schedule 

to account for differences in student aptitude? YES no 

c. Have the interests, educational background, and 

attitudes toward formal schooling been determined? YES no 

d. Is this infomation used to make training presenta- 
tions more meaningful? YES no 

e. Do students receive rewards that are significant to 

them when they achieve course objectives? YES no 

f. Do student rewards include those that are under the 

control of the student company commander? YES no 

g. Has coordination been achieved with the student 
company commander to make rewards under his control 
responsive to student performance in the training. YES no 

h. Are successful students punished? y^s NO 

i. Are failing or borderline students rewarded? yes NO 
j. Has an analysis been made of the amount and reasons 

for excused absences from class? YES no 

k. Have steps been taken to reduce the amount of 

excused absences to a minimum? YES no 

8. Control of the quality of the training. 

a. Are the tests direct translations of the training 
objectives? Y£S no 

b. Is emphasis given to performance tests? YES no 

c. Are grades expressed ia percentage passing? YES no 

d. Are grades based on the bell-shaped normal curve? yes NO 

e. Are grades based on percentile ranks? yes NO 

f . Are test items changed to make them easier or 

harder to conform to an "ideal" distribution of grades? yes NO 

g. Are results of student testing provided to the 
instructional departments? YES no 

h. Do the departments make changes in training pro- 
cedures suggested by the results of student testing? YES no 

(Reference: Smith, Design of Instructional Systems.) 
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to trainees on behaviors they are expected to acquire. 
5. Built-in evaluation of training techniques which re- 
quire trainees to exhibit skills, demonstrate knowledge, 
provide an affective response to training; where pos- 
sible, evaluation instruments should use naturalistic 
and/or unobtrusive observation, 
^es_t. The best possible training program will have, in addition to 
the above: 

6. An explicitly stated rationale for the training program 
that provides statements of: 

. documented need for the training 

. assumptions, theory or evidence that supports the 

instructional approaches used 
. evidence that "real world" job analyses have been 

conducted to derive training objectives. 

7. A history of the development effort which includes pilot 
and field test data and course revision made on the 
basis of trainee performance behavioral data. 

Guidelines for Determining the Quality and Effectiveness of Training Programs 

There are a great many practical outcomes from psychological research 
and training experience that provide valuable information for training 
officers in local, state or federal government. These outcomes have been 
adapted from Tyler (1955) and are presented below in the form of principles 
and characteristics to the training officer as background information. The 
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principles and characteristics may be useful in Judging the quality of 
a) training materials, b) training-in- action, or c) in developing RFP's 
for trainir materials. 

Psychological Principles Useful in Training Materials Development 

1. Learning depends upon motivation. 

2. Learning depends upon the capacity of the trainee. 

3. Learning depends upon the previous e^qperience of the trainee. 

4. Learning depends upon perceiving relevant relationships. 

5. Learning depends upon an active search for meaning on the part 
of the trainee. 

6. Learning depends upon feedback provided to the trainee. 

7. Learning depends upon achieving satisfactory personal and social 
adjustment in the training environment. 

8. Learning founded upon the search for meaning will be more likely 
to be repeated and applied in new settings. 

Characteristics of Effective Training Experiences 

1. Provide the trainee with the opportunity to practice the type 
of behavior he is expected to exhibit on the job. 

2. Provide the trainee with the opportunity to deal with the 
knowledge content implied in training objectives. 

3. Are within the experience range and within the mental 
ability range of the trainee. 

4. Build on the past experience of the trainee. 
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5. Are satisfying to the trainee, 

6. Are perceived as relevant by the trainee in relation to his 
current job and his career aspirations, 

7. Demonstrate to the trainee all of the enabling objectives and 
tasks required to reach the training goal, 

8. Provide feedback to trainees en route regarding achievement of 
objectives so that it can be used in guiding training activity, 

9. Require good personal and social relations. 

Table 4 provides a systematic procedure for judging the quality of a 
training program. Although it is possible for decision-makers to use the 
above procedures in vacuo , it is more effective to include developers and 
evaluators in this feedback pvucr 



serL Table 4 about here 

Contact with Developers 

It may be apparant -.y now that the evaluation of training involves more 

than just determining the aci^jquac; of the training activity. Evaluation must 
be applied at all points of the training system. Thus, if part of the train- 
ing system is subcontracted (e.g., the development of instructional materials 
and activities), it is necessary that those performing evaluation maintain 
liaison with the subcontractors. The role of the evaluator in this instance 
is to provide continuing feedback to the developer regarding the adequacy 
of instructional objectives, in^cructional materials, and instructional 
processes. In this way, the probability of achieving a maximally effective 
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TABLE 4 

How to Judge the Quality of £ Training Program 

Things you must do before you examine existing program manuals and 
materials. 

Step 1. Specify: training population — who Is to be trained? 

length of training In hours — how long? 

general goals of training — what do you want from 

training? 

type of training — what kind of experience do you 
want for trainees? 

cost — how much can you pay? how much can trainee 

pay? 

Step 2. Define what you mean by training: 

Do you intend your training to: yes no 

1. present information 

2. develop r>,7areness 

3. communicate theory/knowledge 

4. promote identification and application of principles 

5. develop an orientation 

6. change attitudes 

7. produce skilled behavior 
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Things you do to judge the quality of training packages you are examining: 

Step 3. Examine and judge the goals and objectives of the training 

package in terms of: acceptable unacceptable 

to you to you 

1. correspondence with your objectives 

2. course length 

3. relevance to your needs 

4. scope of objectives 

5. behavioral quality of objectives 

6. relevance to your pocketbook 



Step 4. Examine and judge the experiences provided for trainees 

Are they related to your objectives? 

Do they require active participation of trainees 

Are experiences varied 

Step 5. Examine and judge built-in evaluation devices 

Are trainees made explicitly aware of outcomes? 

Are trainees given opportunity to evaluate their 
own knowledge of skill acquisition? 

Are trainees provided feedback on their experience? 

Do trainees have opportunity to rate effectiveness 
of training? 

Step 6. Examine and judge background information on 
training program 

Do developers have adequate qualifications? 

Is instructor's role clearly specified? 

Is previous evaluation data presented? 



yes no 
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Seep 7. Summarize your judgments and decide 
Use program as is. 

Seek modifications in the program to make it more congruent with 
your needs. 

Judge it as inadequate for your needs. 
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training package is increased. Parenthetically, it is probable that the 
amount of liaison between de- Topers and evaluators will be dictated by 
the strength of the RFP guiav^iines. The more comprehensive the guidelines, 
the less the amount of liaison between developer and evaluator will be. 

There are, however, critical contacts between the developer and evalua- 
tor which must be maintained. It is necessary, for example, that developers 
and evaluators agree, as soon as possible, on the instructional and beha- 
vioral objectives of the training materials and activities. This interface 
is critical, in that developers must have early feedback regarding the 
appropriateness of their efforts and that evaluators must have sufficient 
lead time to develop procedures for training evaluation. In addition, a 
possible spinoff of this liaison may be the inclusion of evaluation pro- 
cedures as a part of the training activity itself ni very desirable state- 
of-af f a5 rs. 
Criteria Deve l opment 

Coupled with early specification of instructional and behavioral 
objectives is the determination of chat criteria to use as indices of 
training success. Since assessirg the adequacy of behavior for all objec- 
tives vjould be prohibitive in terms ol time and since it would be inordin- 
ately expensive to develop and apply s^ich evaluation procedures, ic is 
necessary to deterirdne what sample of behaviors (and the corresponding 
measurer'ent standard) one is willing to accept as representative of the 
domain cf behavior's that Lraining is focused on changing. The specification 
or criteria at this point is a prelude to tha dev^^lopment of criterion- 
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referenced measures which will serve to help evaluate the effectiveness of 
training. Nitko (1972, p. 6) points out that there are four characteristics 
inherent in criterion referenced tests. These characteristics are: 

(1) The classes of behaviors that define different achieve- 
ment levels are specified as clearly as is possible 
before the test is constructed; 

(2) each behavior class is defined by a set of test situa- 
tions (that is, test items or test tasks) in which the 
behaviors can be displayed in terms of all their im- 
portant nuances; 

(3) given that the classes of behavior have been speci- 
fied and that the test sit ations have been defined, 
a representative sampling plan is designed and used 
to select the test tasks that will appear on any form 
of the test; 

(4) the obtained score must be capable of expressing ob- 
jectively and meaningfully the individual's performance 
character istics in these classes of behavior. 

Glaser (1963) states, "The standard (or criterion) against which a 
student's performance is con5)ared. . » is the behavior which defines each 
point along the achievement continuum (p. 519)." 

"A criterion-referenced test, then, is one that is deliberately con- 
structed to give scores that tell what kinds of behavior individuals with 
those scores can demonstrate (Glaser & Nitko, 1971)." 
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Selected articles have been collected by Popham (1971) which provide 
an overview to the stace of the art of criterion-referenced measures. 
Glaser and Klaus (1962) relate the use of criterion-referenced measures 
to the problem of criteria development. They suggest that: 

The first step in the development of a proficiency measure is 
the specification of the behavior to be observed and measured. The 
ease with which this step can be carried out is dependent upon several 
factors including (1) the complexity of the behavior involved and the 
explicitness with which the behavior has been defined, (2) the purpose 
for which the measurement of behavior has been undertaken, and (3) the 
accessibility of the behavior to observation (p. 430). 
The degree to which successful criteria development is possible rests 
with the adequacy of the task analysis (Miller, 1962). Without clear speci- 
fication of job performance requirements, the development of criteria becomes 
a problem of intuition and guesswork. If this occurs, the best techniques 
available for training evaluation will be incapable of adequately assessing 
the effectiveness of training. 
Techniques for Training Evaluation 

For the most part, the techniques appropriate in most evaluation 
settings can be used in training evaluation. Test formats and item types 
are adequately summarized by Cronbach (1970), Anastasi (1968), Guion (1965), 
Glaser 6c Klans (1962) and Smith (1971). There are, however, some noticable 
differences in item selection and statistical manipulation of data between 
the classical test theory approach to measurement and the measurement of 
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training outcomes. Basically, these differences revolve around the pur- 
poses for which the instruments are to be used. In classical test theory, 
for example, item selection is performed with the major purpose of maximiz- 
ing the variation or the discrimination ability for each item (Nunnally, 
1967). In contrast, evaluating training effectiveness suggests that cri- 
terion-referenced items should be minimally differentiating if the train- 
ing has been effective, e.g., all trainees could conceivably answer all 
items correctly (Popham, 1971). 

Data specification . Central to the issue of data specification is the 
adequacy of the training objectives. Training objectives which are beha- 
viorally stated in measurable terms define what data are to be used in 
assessing training effectiveness. For example, the following training objec- 
tive (Greg/McGraw-Hill and the Center for Vocational and Technical Education 
at the Ohio State University) not only specifies what behavior a trainee 
must be trained in, but it also specifies what class of data must be obtained 
to determine if the trainee has successfully demonstrated such behavior: 
"Given a monthly departmental report of four pages in revised, typed 
draft, with all details complete and organized, the typist executes four 
duplicating stencils (to send to a duplicating department) on an electric 
standard typewriter acceptt^jly within (number) minutes (p. 48)." 

In this case, acceptably may further be defined by the number of typing 
errors allowable (e.g., one error per page). The data which may be used to 
evaluate performance may now be specified in terms of the number of errors, 
the number of minutes, or some ratio of errors to minutes. Data are therefore 
specified in relation to some expected standard of performance and performance 
may be evaluated in these terms. 



Data collection . The collection of data is dependent on a number of 
issues, foremost of which is the degree to which data specification has dic- 
tated what data are to be collected. For example, the above stated objective 
demands that data be collected under the conditions of the objective. 
Therefore, the, data collected will be four stencils typed in (number) minutes 
on an electric standard typewriter. In this case, a formal test may be 
given (obtrusive measures) or a typist •s daily training records may be re- 
viewed (unobtrusive measure). Data for these conditions can be obtained 
from training records which indicate at what performance level a typist is 
functioning. Thus, unobtrusive evaluation can be built into the training 
activity by merely keeping training performance process records. 

Other issues underlying the data collection problem are summarized 
in Table 5 in which the data collection mode, the advantages and disad- 
vantages of that mode, and a conclusior. regarding the use of that mode of 
data collection are displayed. Parts of Table 5 are adapted from Frederiksen, 
in Glaser (1962) . 

Insert Table 5 about here 

Data processing . Although most data processing can be performed ade- 
quately by hand, the disadvantages far outweigh the advantages. A more 
efficient, flexible, and accurate method of data processing can be found in 
the use of automatic equipment such as key punch machines, card sorters, and 
computers. The advantages of machine use for data processing begin with 
accuracy. For example, key punching and verifying data directly from process 
records or evaluation test forms can be expected to result in less than 
one data error for every 1000 datum recorded. Usually the error rate is 
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EVALUATION MEASURES—ADVANTAGES & DISADVANTAGES 



Approaches for Obtaining 
Evaluation Measures 



Advantages 



Disadvantages 



Conclusion 



1. 



Student — 2. 



3. 



1. Solicit Opinions 



Highly reliable 
judgements about 
teaching can be 
made. 

Useful for learn- 
ing student's o- 
pinions. 
Easily done. 



1. High reliability 
is in part a re- 
sult of student 
pre-discussion, 
which destroys 
specificity. 

2. Evaluates in- 
struction, not 
job performance. 

3. Students are in- 
clined to say 
favorable things. 

4. Students may not 
be good judges 
of performance. 



As a way of eva 
uating success 
in achieving 
training objec- 
tives, the meth 
has little to 
commend it exce 
the ease with 
which it can be 
done. 



Expert — 1. Useful to an in- 1. Evaluates in- 



structor, to cor- 
rect presentation 
faults. 



struction, not 
job performancec 
Lack of specifi- 
city. 



Misses the pay- 
off question; t< 
what extent has 
behavior been 
modified 



2. Administer Attitude 
Scales 



Obvious approach 
to find out if 
student has a- 
dopted desired 
attitudes is to 
ask the student:. 
Easily done. 



1. Bias due to re- 
sponse set. 

2. Evaluator is in- 
terested, not in 
attitudes, but 

in behavior changes 
presumably rela- 
ted to attitude. 

3. Lack of specificity. 



We rarely have 
evidence that a 
score on an attJ 
tude scale is 
correlated with 
behavior to any 
marked degree. 



*Table 5 summarizes Frederiksen's (1962) analysis of proficiency tests for training 
evaluation. 
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Measure Knowledge 



!• appropriate to 
assessing re- 
sults of know- 
ledge training, 

2. Easily done. 



1. 






Real objectives 


To the extent 




have to do with 


that the desired 




behaviors other 


objectives of 




than knowledge 


training have to 




ability. 


do with performance 


2. 


Ability to per- 


it is better to 




form a task is 


use measures of 




not always rela- 


that performance 




ted to knowledge 


as the basis for 




of related facts 


evaluation. 




and principles. 






Lack of specifi- 






city in all areas 






of training out- 






comes except 






knowledge. 





Elicit Related 
Behavior 



1. Better than 
approaches 1, 2 
and 3 because 
attempts to ob- 
serve behavior. 

2. Easier than e- 
valuating beha- 
vior which more 
closely approxi- 
mates the ulti- 
mate criterion. 



1. Validity of cri- 
terion cannot be 
taken for granted 
because relation- 
ship of the rela- 
ted behavior to 
the ultimate cri- 
terion is inferred 
on the basis of 
logical relation- 
ships. 



Stronger than 
approaches 1, 2 
and 3 because the 
evaluator attempts 
to measure beha- 
vior. Measures 
behavior less 
directly than ap- 
proaches 5, 6 
and 7. 



Elicit "What I 
Would Do" 
Behavior 



1. 



Better than ap- 


1. 


In multiple choice 


There is only 


proaches 1, 2 




versions the trainee 


one real 


and 3 because 




doesn't in. ..t solu- 


answer to sit- 


the evaluator 




tion:, ^hc OSes 


uational items, 


does observe 




them. 


"It depends," 


behavior of the 


2. 


Test-taking behavior 




type students 




can represent an at- 




are being 




tempt to choose a 




trained to per- 




"correct" answer, not 




form. 




what the examinee 




Easier than 




would do. 




eliciting or 


3. 


Situations are too 




evaluating be- 




briefly described 




havior which 




to present "real" 




more closely 




problems. 




approximates 




May fail to get a 




the ultimate 




"style" of behavior 




criterion of 




displayed in real 




real-life be- 




life. 




havior . 
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6. Elicit Lifelike 
Behavior 



1. Examinee be- 
haves as if he 
is in a "real- 
life'* situation 
calling for be- 
havior of the sort 
he has been train- 
ed to perform. 

2 . S tandardized the 
"real-life" sit- 
uatior . 

3. Face validity is 
high if simula- 
tion is done 
well, 

4. Method can mea- 
sure objectives 
of attitudes 
stated in terms 
of behavior, not 
in self-report 
terms. 

5. Allows evaluator 
to control his 
criterion var- 
iables. 

6. A behaviorally re- 
lated attitude 
variables can be 
successfully em- 
bedded in a total 
performance test . 
Easier than ob- 
serving "real- 
life" situation. 



1. The examines muHt 
not know the at- 
tributes of atti- 
tude to be scored 
if attitude mea- 
surement is part 
of the total per- 
formance test. 
Hence the approach 
is not effective 
for continued 
routine measure- 
ment of attitude. 

2. More difficult 
than methods 1 
through 5. 



'\'\\v. lypi* ol nuNi- 
8ur e r ec ommcMul ed 
for first con- 
sideration in a 
training evalua- 
tion study is 
that which most 
nearly approximat 
a real-life situa 
tion, the "elici- 
ting life-like 
behavior" approac 
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Observe Real-Life 
Behavior 



1. This approach 
is the ideal 
criterion 
measure. Its 
validity can 
not be ques- 
tioned. 



1. Test conditions 
are not standar- 
dized. 

2. Occurrences of 
behavior measured 
may be rare. 

3. Criterion beha- 
vior may be 
private. 

4. Costly in time 
and effort. 

5. It may be hard 
to select and 
analyze rele- 
vant behavior 
aspects from 
all the matrix 
of behaviors. 

6. Rating scales 
are often 
used because 
performance 
events are dif- 
ficult to 
scale. 



Psychometric 
problems are 
insurmountable . 
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much less. A random sample of hand transcribed data will reveal a much 
higher error rate. 

Once data have been recorded (keypunched) , manipulation of data by 
card so: ter or computer allows for an almost luilimited number of data com- 
parisons. Having this flexibility allows for efficient data summarization. 
Further, raw data can be stored for aggragation when additional data on the 
same variables are collected. Thus, data can be used as process information 
in a comparative sense. For example, data may be obtained from a number of 
typing training courses offered over time. A comparison of data collected 
during each training course will provide evidence of specific course effec- 
tiveness, Lhereby yielding feedback capable of being used for course modi- 
fication. 

A final advantage of machine processing of data is that the analysis 
of data may be efficiently accomplished drawing from a wide spectrum of 
statistical procedures. A good example of statistical procedures available 
on computers may be found in Veldman (1967). 

Data analysis . Orce the data are collected an'', processed, statistical 
analyses of the data may be performed. The nature of the statistical tech- 
nique used to manipulate data is primarily dependent on two things: (1) 
the nature of the data, collected, and (2) the questions being asked of thn 
data. A good summary of assumptions underlying the most popular statis- 
tical techniques may be found in Guilford (1965). 

The questions which are asked of training data are usually related to 
the adequacy of the training. Thus, questions such r.i' -'Did the training 
accoii5>lish the desired objective?" help to determine, in part, what statistical 
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test to apply to the data. If a training standard has been specified (e.g., 
80 percent of the trainees will correctly respond to 80 percent of the 
measures), then the only statistical treatment necessary would be that of 
determining the percentage for each of the measures used. This is usually 
the case if criterion- referenced measures are employed. Since criterion- 
referenced measures are not designed to maximize variation, the use of in- 
ferential techniques, such as the analysis of variance (which have test 
variation as a basic assumption) , would not be appropriate. Although 
there are some exceptions, in general, the use of most statistical techniques 
based on classical test theory are inappropriate with criterion- referenced 
measures. An adequate discussion of this point is presented in Popham 
(1971) and Nitko (1972). 

Data reporting . The one major requirement of data reporting is that 
the data be summarized in such form so that any reader can immediately re- 
late the data to the questions posed. It is important, therefore, that data 
be relevant, concise, and clearly displayed. To satisfy this requirement, 
users of data should help to determine what their needs will be. 

Community Development Training — The State of the Art 

Assumptions 

General acceptance : :is been afforded the concept that the basis for 
training and education programs rests in adequate ideiicif ication of needs 
or by performing needs asses i t. Thus, the Survey of Training Needs and 
Evaluation of the 1970-1971 : -^.e VIII Training Program (Texas Municipal 
League, 1972) served to perform this function for the State of Texas. One 
objective of the report was "...identifying training needs in Texas muni- 
cipalities (p. 1)." The conclusion of the report is that "justification 
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for training is self-evident (p. 1)." The rationale for this conclusion 
includes: 

1) The impetus for expanded training programs for Texas 
municipalities is found in the needs of the State's 
toxms and cities. Total population growth and 
changes in population patterns are indicative of 
burgeoning urban needs and problems (p. 2). 

2) ResDonse to urban issues has been multi-faceted, 
and one facet of this response is heightened em- 
phasis on upgrading the skills of public policy- 
makers, managers, technicians, and workers (p. 2). 

The primary data which support this rationale are responses to a survey 
designed to identify specific -raining needs for municipal officials. These 
data are forecasts of training needs by previous course participants and 
city officials, A summary of the specific findings and recommendations 
made by TML f o 1 lows : 

Inadequate post-entry training is a stumbling block to effective 
municipal government in Texas. The problem has stemmed from limited 
budgets at the local level and unchanging resources for a cohesive, 
coordinated effort to formulate effective statewide programs that are 
tailored to identified problems. 

Municipal government growth. . .will accelerate an already serious 
problem. More and better educated administrative, professional, and 
technical (APT) personnel will be required for city operations, and much 
greater demands will be made for keeping the skills of APT employees in 
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middle and upper echelon positions current* The rapid changes in 
techniques brought about by the application of increasingly advanced 
technology lends even greater importance to the latter point. 

Needs vary from city to city, depending on size, the peculiarities 
of the local economy, level of services offered, and other factors. The 
dominant aspect of differential needs, as related to the rudiments of 
APT functions, is size. The degree of specialization in personnel skills 
accelerates with the size of city population and government structure, 
and the scope and diversity of government functions. 

Smaller cities must depend on generalists - city secretaries and 
managers backed by little or no administrative, professional, or technical 
staff. Budget limitations and the practical job requirements for "minding 
the store'' inhibit participation in extensive and expensive training 
programs held in distant locations. Alternative information channels, 
such as professional associations, are a useful source for upgrading 
skills, but these sources are usually pointed toward the esoteric technic ^ 
of specialized subject matter and are often not directly transferable co 
small city needs. 

At the other end of the spectrum, large cities are as complex as 
any institution and must have access to the highest level of skills pre- 
vailing in the state of the art of any subject matter or techniques 
related to the provision of municipal services (pp. 12-13). 

Apart from the obvious subject matter priorities ... the most salient 
feature of the analysis of future training needs is the v riance in 
priorities assessed the various topics by participants from the three 
population brackets in which data were aggregated. The survey seems to 
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confirm Phase I survey findings that the immediate needs of smaller cities, 
particularly as related to APT personnel, include generalj'st skills in 
management and administration • 

On the other hand, larger cities indicate needs for more sophisti- 
cated training in areas that have only recencly evolved as concerns for 
municipal government. Housing needs, community development, human 
resources and urban technology are all illustrative of emergent trends in 
cities as opposed to the more traditional functions and services conducted 
in metropolitan areas. APT functions which might be considered part of 
traditional operations of cities were given low ratings, probably because 
most of the larger cities have developed a high degree of specialization 
and expertise in such areas as personnel management, general management 
and operations, public health programs, etc. 

The major inference which may be drawn from forecast findings is 
that constant attention must be given to designing courses for different 
sizes and types of municipalities. The approved 1971-72 TML training 
program already reflects differential needs to some degree. An urban 
technology seminar and non-profit housing course oriented toward the 
larger cities is planned, and courses in small city administration and 
budget preparation will be conducted for the smaller municipalities. The 
remaining programs - code enforcement, community development seminar and 
case problems - cut across the board and will serve the documented needs 
of all cities. Future evaluation of training programs will be formulated 
to refine even further the distinctions in needs among different sized 
and types of municipalities (pp. 25-26). 
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The needs assessment suMaaary reported by TML, presented abcve, does not 
meet the requirements of describing behavior discrepancies bet-ween how APT 
personnel are actually performing and an ideal 'ev<?.l of performance. These 
requirements form one aspect of the DOES modrwl and are best sumarized by 
i.aufman (1972). 

He points out that there are three necessary characteristics of a needs 
assessment. These are; 

1) The data must represent the actual world of learners 
and related people, both as it exists now and as it 
will, could, and should exist in the future. 

2) No needs determination is final and complete; we must 
realize that any statement of needs is in faLt tentative, 
and we should constantly question the validity of our 
needs statements. 

3) The discrepancies should be identified in terms of pro- 
ducts or actual behaviors (ends), not in terms of pro- 
cesses (or means) (p. 29). 

Although TML alluded to characteristic number 2 in their needs assess- 
ment report, the major focus of their study seemed to be aimed at identifying, 
at a general level, the areas for which training might be appropriate. Clearly 
TML does not meet the criteria presented above in their needs assessment study. 
constraints 

As indicated above, a number of constraints are associated with Texas 
municipal training programs. Specifically, the general training needs vary 
according to size of community. In addition, the amount c : "work hours 
training time" allowed is small, and also varies with che size of the community. 
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In general, the situation demands that training efforts be short and directed 
at the awareness level. 
Levels of Training 

If training courses developed for the State of Texas are based on 
TML's needs assessment, that information provided by the needs assessment is 
not specific enough to allow for the development of skill or knowledge level 
training. Although TML specifies that APT personnel need skill training 
there are no specific discrepancies reported which would warrant this recommen- 
dation. Further, the data presented by TML are not a form that would enable 
developers of training to specify adequate training goals and objectives. 
Goals of Training . 

One of the major difficulties in formulating adequate training goals 
rests with the adequacy of needs assessment. Thus, the more general the 
needs assessment, the more general the goals statements tend to be. The 
recommendations made by TML are formulated on the basis of global data. 
These data, then, are not consistent with the stated goals of training, e.g., 
to provide skill training. As a result, the following three problems emerge: 

1) Inadequate specifications of training objectives which 
are focused on producing observable behavioral changes; 

2) Inadequate design of training programs wiiich are 
capable of producing desired behavioral change; and 

3) Inadequate criteria (standards/objectives) against 
which the success of training may be judged. 
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Qbiectives of Training 

As such, formalized statements of training objectives are rarely 
found in Texas training courses. Most generally, when objectives are stated, 
they appear in the form of instructional objectives (Gronlund, 1970). An 
instructional objective is a general statement of which a class of behavioral 
objectives are a part. For the training of specific skills, instructional 
objectives stop somewhat short of specifying desired outcomes against vhich 
the success of training may be evaluated. Objectives must be stated so that 
the expected action or behavior of a trainee is specified along with the 
conditions under which the behavior will be performed and the standard of 
acceptab le performance . 

Statemea: of Need for the Evaluation of Training 

Although the design of most training courses provides for some subjective 
or impressionistic evaluation, there exists a persistent need for performing 
evaluation focused on determining the adequacy with which goals and objectives 
are being achieved. In this respect evaluation must be concerned with more 
than a specific course. Adequate and effective evaluation is predicted on 
viewing training as only a small part of a larger system. Thus, for evaluation 
to be effective, all aspects of the system including comprehensive needs 
assessment, specification of institutional goals, analysis of job performance 
requirements, etc. must be related logically before evaluation of any part 
of that system can be performed. Once congruence arnong all elements of the 
system has been determined, the evaluation of training can take its logical 
and necessary position. 
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Summary 

The purpose of this section has been to show the relationships between 
training and evaluation as reflected by the Decision Oriented Evaluation System. 
An overview of the state-of-the-art of training and evaluation revealed an 
increase in the level of sophistication in these areas. The amount of research 
and development in training and evaluation hvas increased rapidly in the past 
few years as is evidenced by substantial govenainent spending. Yet it has been 
pointed out that much research is still needed to provide training and evalua- 
tion personnel an effective methodology with wtiich to accomplish their goals. 

The Decision Oriented Evaluation System is proffered for consideration 
as a tool in achieving training and evaluation goals. The system emphasizes 

1 importance of specifying goals and objectives in ob^^irvabie terms based on 
systematic needs assessment and job analysis data. Further, the desirability 
of equivalencing levels of training to levels of evaluation is expressed. 

A general trailing model (developed by the Air Force) and a general 
instructional system are presented along with guidelines to facilitate deter- 
mining the effectiveness of an instructional system. In addition, specific 
procedures for evaluating training, inducing the advantages and disadvantages 
of these procedures, are presented. 

In the last part of this section, au attempt was made to describe the 
state-of-the-art of community development trainicjg. The desirability of 
parfoming adequate needs assessment was emphasized along with conclusions 
regarding the constraints, goals, objectives, and evaluation of training in 
state supported programs. 
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Cioncluslon and Recomrendatlons 
Although the scope of an entire evaluati system far outreaches the 
scope of training evaluation, there is little substantive difference in the 
procedures necessary for effective and adequate operations. In each case. 



and objectives are being achieved. The data used to provide this evidence 
are derived from making observations of behavior. From these observations, 
inferences regarding better or different methods for reaching specified 
goals or obiectives may be made. Thus, evaluation is a process of identifying, 
obtaining, and using information to judge decision alternatives and an 
evaluation system is a set of procedt^res which provide valid and practical 
information to decision- makers for making program decisions and judging 
goal attainment. 

It can be assumed that there is a necessary connection between needs 
assessment, training, and evaluation (as well as job performance and com- 
munity impact) and that these elements form a system. One of the fundamental 
laws of a system is that it is only as strong as its weakest element. In a 
training and evaluation system, then, a weakness in any aspect of that system 
will have effect on the rest of the system. These points need to be con- 
sidered, not only when systems are beinj^ defined and developed, but also as 
the system is operating. 

T.\e sLite of the art of both training and evali: tion has not yet reached 
a levd of precision necessary for unqualified r ^omm..ndations. This is par- 
ticularly true of a training and evaluation system. There are, however, some 
basic principles which can serve as anchor points for sound evaluation. 
These principles are: 



evaluation is focused on determining the degree to which specified goals 
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(1) A needs assessment is the cornerstone of training and 
should provide information regarding the discrepancy 
between the "way things are" and "the way things should 
be". 

(2) Goals for training should be derived from needs assess- 
ment information and should be stated in measurable terms • 

(3) Instructional objectives and behavioral objectives should 
be derived from goal statements and should be stated in 
meas urable terms • 

(4) Evaluation of training should focus on the degree to which 
•goals and objectives are being achieved. 

(5) The discrepancy between actual achievement in a training 
situation and the specified objectives of that training 
will help form the needs assessment information base for 
further training development. 

(6) The discrepancy between actual job performance level and 
desired job performance level will add needs assessment 
information for training development objectives. 

(7) The desired job performance level should be specified 
in behavioral terms based on job task analysis. 

(8) Job tasks should be related to and consistent with the 
goals of the organization in which they are performed. 

(9) The goals and objectives of the organization should be 
stated in measurable terms and should be related to and 
consistent with the needs of the population the organi- 
zation serves. 

o 79 
ERIC 
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(10) All elements of a train-ac-: and evaluation system, including 
needs assessment, training development and evaluation, job 
performance evaluation, job task analysis and community 
impact evaluation must be intiernally consistent in terms of 
goals, objectives, and job performance. 

(11) All goals and objectives should be stated in measurable and 
attainable terms . 

(12) Developers and evaluators of training should maintain close 
liaison throughout the training development and activity 
period. 

(13) Evaluation should be designed into the training activity as a 
natural and preferably an unobtrusive event. 
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GATE Specific Implementation of Training Evaluation 
Backgrou nd 

The procedures employed in implementing training evaluation for the 
Texas Department of Community Affairs began, initially, with an extensive 
specification and evaluation of the assumptions underlying Texas Department 
of Community Affairs Training activities. The specific model used in this 
effort was derived from Stuff lebeam (1972), to which all elements of the 
training program we- related. The following were considered major elements 
to be considered in the evaluation of training: (1) target population; 
(2) level of training; (3) length of training; (A) assumptions; (5) con- 
straints; and (6) contact with developers. The training activity which was 
chosen to pilot test the Decision Oriented Evaluation System was '^Principles 
of Supsrvision" , developed and executed by the Engineering Extension Service 
at Texas A&M Unxversity. 

Target population . The target population of the training course is 
basically "first-line" supervisors. Supervisors, from whom data were 
collected at the end of the training course, tended to fall in this category. 
Analysis of background data revealed that the mean age of the 21 participants 
was 3A. 5 years. Fifteen trainees were males and 6 were females. On the 
average trainees reported '^at they had been a supervisor for 5 years and 
they supervised, on the average, lA people. Thirteen trainees reported 
that they were "office" supervisors and 6 trainees indicated that they 
supervised shop personnel. Most of the trainees (15) reported that they 
had attended at least one other supervisory training course. Six reported 
that they had attended no other supervisory training course. 

Er|c 81- 
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l.evoi of Lrainin^ . An evaluation of training materials revealed that 
this counit: w'l.s ba;: I ca i 1 y an awari*rH»H.s hu I 1 d J ng-or I (?n t a I I on type? oT activity. 
The focus ol training revolved around the use of a "Discussionaire" which 
served to act as a topical guide for discussion by trainees. The material 
presented in the "Discuss ionaire" theoretically focused on he major principles 
and job functions of supervision with "adjustable" content so that the course 
would be able to adapt itself to the operation and needs of the group 
participating . 

Length of training . "Principles of Supervision" is one of s. supervisor 
training courses in Texas A&M University's Supervisory Development Program. 
Each course is described as a 15-hour course calculated provide 90 hours 
of intensive management and supervisory training. Although it is recommended 
that the course be given for two hours a day over a work week period, this 
particular course was given in two consecutive da; i for about the same total 
number of hours. 

Assumptions . One of the basic assumptions initially held about this 
course was that the training objectives were derived from job analyses 
resulting in sKill development; however, further information revealed 
that the course would only provide information and orientation about Principles 
of Supervision. A second assumption, based on irfo^mation provided by developers 
based on their past experience, was that trainees involved in courses of 
this nature would have a relatively low education level. This proved not 
to be the case in the pilot test. A third assumption underlying the evaluation 

design was that all topical areas in the "Discussionaire" would be covered 

by the course. This did not occur. 

Constraints . The major constraints impinging on the evaluation of 

the "Principles of Supervision" course stemmed from two sources: (1) a 

ERIC 
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lack of clear specificity of training goals and objectives; and (2) a lack 
of complete training materials and guidelines related to course activities. 
These two features coupled with the fact that skill training (although this 
was not the objective of the course) was not provided, limited the effective 
ness and adequacy of the subsequent evaluation design. 

Contact with developers . At all points during the design of the evalua 
tion, close contact was maintained with members of the training staff at 
Texas A&M University. The final evaluation design was based on the efforts 
of the developers and evaluators in a collaborative relationship. The fol- 
lowing major issues were identified and resolved as a function of this rela- 
tionship: (1) identification and specification of course goals and objec- 
tives; (2) criteria selection; (3) item development; and (4) application of 
items. The following is the interpretation of objectives for Supervisory 
Training Program #1, '^Principles of Supervision". 

(1) Goals and Ob j ec tives -- Identif icat ion and Specification 
Cognitive Outcomes 

1. The trainee will demonstrate his understanding of 
the broad role of the supervisor as it relates to 
manii^ement and workers. 

2. The trainee will demonstrate his understanding of 
i:he basic principles of management and supervision. 

3. The trainee will demonstrate his ability tc apply 
principles of management and supervision ia a 

' "simulated" \7ork setting. 
Affective Outcomes 

4. The trainee will demonstrate a preference lor the 
role of the supervisor as presented in the course 

83 



(see objectives 1;, 2.. 3). 
5. The trainee v;ilJ. express a desice to improve hjn- 
self as a supervisor and will take action vUat 
end (see objectives 1, 2, 3). 
D_ q^£iaition of Terms 

A. Understand ing: 

Objective #1 - ability to discriminate among 
relevaJ r and non-relevant items related to 
super\;icJor's role. 

Objective ?r2 - abi!'!,.;- io differentiate among 
duties of supervisor manager in terms of 
r a rJ<. ed p rl o vi ty . 
b. Apply. 

0'>jectivc: ~ abil.Tty to select an appropriate 
priDciplca and indicate appropriate point of 
app L Lcat iun. 

C. Preference: 

Objcictive ?M - expresses statement indicating 
that hv> values the position of the developer. 

D. Desire: 

Objective #5 - seeks cut materials to improve 

self a5 supervisor. 
Evaluation Criteria for Objectives of the Supervisory 
Training Program - Principles of Supervision 
Principles of Supervision Items 

Given a list of 14 principles of management and 
supeirvision, 807o of the trainees will select (identify) 
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correctly at least 70% (10) of thoci principles 
which are basic or irrelevant to supervision. 

Correct identification is defined as the 
number checked correctly plus the number left 
unchecked correctly. A criteria of 70/1 is 
equivalent to 10 correct choices out of 14 pos- 
sible. 
Job Functions Items 

Given a list of 22 job functions relating to 
(management), supervision production, etc., 807o 
of the trainees will select (identify) correctly 
at least 70% (15) of those functions related or 
unrelated to the broad scope of supervisory function. 

Correct identification is defined as the number 
checked correctly plus the number left unchecked 
correctly. A criterion of 70%, is equivalent to 
15 correct choices out of 22 possible. 
Application of Principles in Situations Items 

Given brief .scriptio" s of six situations 
requiring the ap^>iication of principles of 
supervision and management, 80%, of the trainees 
will be able to select (identify) from a list of 
8 to 10 alternatives for each situation at least 70% 
(40) of the most effective, least effective, or 
neither effective or ineffective ways of lealing 
with the situations 

85 



79 



Correct identif i^^ation is defined as the 
number checked corr'^tly plus the number left 
unchecked correctly. Minimum score is 9 (167o); 
maximum score is 57. A criterion of IQX is equiva- 
lent to 40 correct choices out of 57 possi" e. 
(When test data were processed, the correlations between the scoring 
procedure outlined above and the traditional procedure of scoring correct 
responses only were: (1) 0.23 for the sum of Principles of Supervision 
Items, (2) 0.82 fox the sum of Job Function Items, and (3) 0.87 for the sum 
of the six Situation Items. The low correlation for Principles of Super- 
vision Items can be attributed to a lack of variation in the items - typical 
of a criterion referenced measure.) 
Pilot Test Development 

After agreement was reached by Texas A6^ and GATE staff members on 
the Principles of Supervision workshop objectives, items related to these 
objectives were developed. The "Discus sionaire" used in the workshop formed 
the primary content base from which each item was derived. A second content 
source was the course outline. 

Item d uivelopment . Based on an agreed upon format, items were developed 
for three eval ition sections* (1) Principles of supervision, (2) Job 
functions, and (3) Application of supervision principles in situations. 
These it'-^-ns, in their original form, are di5:played in Appendix E. Texas 
A6cM staff members reviewed the items and suggested '^at che ' readability" 
level may not be appropriate for a typical population of worksh p trainees. 

Subsequently, each section of the evaluation instrument was analyzed 
for readability level using the L^le-Chall readability formula (1948a, 1948b). 
The work sheet u::;ed to de ermine readability is displayed in Appendix F. 

i 
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The results of the analysis v. Ji ated that items for: (1) Situations 1, 
2, 4, and 6 were inappropr Lai e heiow the 7th grade level of reading; 
(2) the demographic iuf oriiia lIoii and Situation 5 were inappropriate below 
the 9th grade level; (3) i ..inciples of Supervision items and Situatrion 3 
were inappropriate below 11th grade leve .; and (4) Job Functio/. .r^; in- 
appropriate for othr than college graduates. 

Using the readability analysis as a guide, the evaluation instrument 
was revised so that each section and situation had a readability level below 
the 9th grade. It was assumed that some of the job-related" words con- 
sidered to be relatively difficult for a general population would be under- 
stood by supervisors attending the workshop. The final vers a of the 
Principles of Supervision evaluation instrument is displayed in Appendix G. 

Correct choice identification . To determine the correct choices for 
items in each of the sections of the evaluation instrument, instructors of 
che Principles of Supervision course were asked to independently judge the 
correct choices. Seven instructors participated in making those judgments. 

For the Principles of Superv^ision items, 6 of the 7 selected Items 
2 and 13 as the most important principles. All 7 of the judges did not 
::heck frems 5, 8, 10, 12, and 14. The rest of the items were selected 
by 1 to v.»'iges. Items 2 and 13 were selected to represent correct choices 
for thif s^ct:.on o^ the instrument. 

F,r iie section on Job Functions, at least 5 out of 7 judges selected the 
following -nccior.iS ^ thos^^ job.s a supervisor should do: Items 1, 

2, 3, 4, 3> 6, 17, 18, 19, 20, ana 22. These items were selected as cor- 
r^ "It responses. All 7 raters did not make responses to Items 7, 9, 10, 12, 
13, 14, 15, and 16. Fvom 1 to 3 judges checked Items 8, 11, and 21. 
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For * -hird sc.rtion of the evaluation instrument, Situations 1 throuf/.h 
6, the mai. 'a Table 6 summarizes t:>.e number of judges selecting 
the po:, ■ t; and negative alternatives for each of the situatims. 

Insert Table 6 about here 

The degree to which the Texas A&M staff agreed on the importance of the 
Principles of Supervision and the Job Functions items dicates that there is 
relatively high intt^r-judge agreement regarding the correct choices for 
these items. It would appear then that these two seccious of the evaluation 
instrument have at least conten" (or face) validity. This is not the case 
with the situations. As can be seen from the matrix of responses in Table 
6, agreement was obtained on only 3 of the 6 situations (#2, #3, and #5). 
In Situations #1, #4, and #6 it Cc^n seen that there is high agreement 
on 3 of the alternatives. The rest of the alternatives are about equally 
desirable or und esirable» On the basis of the above results, it was con* 
eluded that correct responses for an item would be defined in terms of the 
number of judges selecting that item. An item was defined as a correct 
choice if 4 or more judges selected that item as a correct choice. 
Analysis of Data Obtained fr o ^ : the City of Austin Supervisors 

Data for 21 protocols were coded and key-punched. Using the Texas A&M 
aff responses as indicators of correct resporises , each of the traineeo ' 
instruments were sco ed for 3 variables. Variable #1 is th*? sum of correct 
responses for the Principles of Super^isioi, variable #2 would be the sum 
of correct responses for Job Functions, and variable #3 would be the cor- 
rect 'esponses for the 6 Situations. 

Correlation an -lysis . The 3 instrument vr l iable., were correlated wi'h 
the o background variables, and 6 of the intercorrelations were significant: 
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1. Older trainees tended to have more years of supervisory 
experience (r = .42; P<.05). 

2. Females tend to supervise more people than males 
(r = .54; P<.01). 

3. Females tend to supervise White Collar work and males 
tend to supervise Blue Collar work (r = -.46; P<..05). 

4. Supervisors of White Collar work tend to n^v^e had more 
supervisory courses than supervisors of Bi.ue Collar 
work (r = -.52; P<.05). 

5. Blue C. Tlar sup',rvisors t.jnd to have a higher score on 
Job Functions than Whit. Collar supervisors (r = .56; 
P<.01). 

6. Trainees who scored high on Principles of Supervision 
tended to score low on the six Sitr.ttions (r = -.43; 
P<.05). 

Evaluation of course objectives attainment: . In addition to the corre- 
latioi: .n:..lysis, frequcr.cy distributions were obtained for trainees clvcain- 
ing a 'en Raw Score (see Table 7). At leapt- 80% of the trainees responded 
correctx_- to at l?ast 707c of the iteii:«s for bt :h Pr::.nciples oi ?-'pervision 
and Job Functions. This was not true for the Situations; only 52% of the 
crainces responded C''>rTreL:^ly to at least 70% of the items. 

Conclusio - 3. In t:;tn.^ral, responses to the Principles of Supervision and 
Job Functions sections o, the evaiitation i; ' rumen t ind icated , that the 
workshop adequately covered these V.opici^. , The major problem noted for 
these sections wa^ ^h^^t there were only two correct itcias 1 rMed 
on th: ^rii ciples of Supervision items, suggesting that an adequate sampling 
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of items relevant for this domain was not made. It is notable, however, 
that trainees were able to correctly discriminate on the items for these 
first two sections of the evaluation instrument. 

The degree to which trainees were given the skills to apply their 
knowledge (at least in a paper and pencil simulation as reflected by the 
six situations), however, appears to be less than desirable. If the evalua- 
tion criteria fc?r the obie«:::: os of the workshop are adequate, then it 
might be concluded that either the six situations were not related to the 
material presented in the workshop, or the correct choices for the situation 
items were iTJappropriate . 
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App l IcaLiori ul Decision Information Surnmar:^ to Pilot: Hem Data 
Based on the data obtained for the Principles of Supervision 
workshop, judgments regarding the training we e made and are dis- 
played in the Decis ion Information Summary Trainir;;^ Process Evalua- 

^on ( Fig ure 4 ). The display of these judgments, based on 
available data and observation of che course, give rise to the fol- 
lowing hypotheses: (i) The test instrument may have been inadequate 
for measuring course outcomes; (2) the time and conditions of 
administration of che test instrument may have negatively influenced 
trainee performance; (3) the course may not ha^ , adequately dealt 
with the written objectives and the cest items; (4) there may have 
been insufficient opportunity for trainees to practice the behaviors 
required by the test instrument. The conclusion reached is that any 
or all of the following co:.-rse objectives, course delivery, test 
instrument and test administration should be modified and re- 
evaluated before final judgments are placed o the efficacy of the 
course or the evaluation proced ;res. 

The hypotheses noted above are derived, in part, from the fol- 
lowing information: Fir^c of all, it needs lo be pointed out that the 
Principles of Supervision course is the first in a sequence of six 
courses de^^^ned .or experienced ad/or novice supervisors. Final 
judgment on the el'i'cacy of thi courses really should await administra- 
tion of the ent ire sequence to the same !:arget popu lat i.oa . Only '.n 
that way can the ultimate objectives of tae course be verified. 
Secondly, the objectives ' ved for the course may not have directly 
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corresponded to the objectives of the instructor; although tho parti- 
cular instructor did participate in the verification of the initial 
objectives of the course, we do not have any evidence that he acLualiy 
subscribed to the final objectives and/or the test items. Three, the 
training populat ion for this cour se was at yp ic al of the population for 
whom the t^s^ instru:nents were developed „ The test items were 
particularly directed toward pi edominantly male, public utility 
public service oriented supervisors with a maximum of a high school 
education and sometimes less. It is important to note that the test 
items themsel es were written not to exceed a ninth grade level of 
reading ability. It was also impressed upon «-he test developers that 
many of the trainees attending the course would undoubtedly be unable 
to read or write . 

The actual Lrainees for the course were atypical from the popu- 
lation. expected based o^i the experience of the course instructors. 
First of all it is important to note that 25% of the trainees were 
women. Sixty-six percent of the trainees were employed in social 
service agencie , largely funded from federal sources, and dealing 
with far different situations than those presented in the case study 
type items. It is further important to note that more tihan 50% of the 
trainees were at least college gr duatds. Several of them had graduate 
training. The point to be made very clearly is that the test was 
developed for r far different popula :ion than that which actually 
attended the course. The nature of the test, the nature ;f the delivery 
of the course may in some way account for the performance noted. 
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It should also be pointed out that the test administration con- 
ditions were not optimal. First oi all, the test was added on to the 
natural schedule for the course, and therefore consumed twenty-two 
minutes of course tim that would ordinarily have been directed 
toward the course objecciv(::;> , Some of the actual activities planned 
for the course were not: cr aducted due to the press of Lime; this may 
have affected trainee performance on the test instrument, 

Oi" more particular importance to test results was thn fact that 
the r^iSt was administerc:Q orally by the instructor. That is, he 
read each item and each response for the trainees. la order no complete 
the test in a reasonable amount of time, the instructor read the items 
very rapidly. At one point during the course of the administration, 
one of the highly educated female members attending the course asked 
if the instructor would please net read the items and let the trainees 
read and think about them. She noted th?' she was being too hurried 
in having to respond to the test items and was not able to chink abcit 
the responses to uae test and further than the instructor's reading 
of the items were interfering with her thinking processes. At that 
poi. th2 instructor asked the entire group if he should continue in 
the manner L :v as pursuing or whether he should stop and let the 
train-OS read and respond to the items themselves. The instructor 
asked for a show of hands and slightly more than 50% of the ^ainees 
indicated that they wanted to continue with the test administration 
und« the conditions that were being followed. This mode of test pre- 
sentation Clearly disturbed se veral of the stuJents. The observers 
seatc in the back f the room were a! le to deiect observable body 
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behavior which indicated dissatisfaction with the way in which the test 
was being administered. 

In summary, it is premature to place any final judgment on the 
effectiveness of the first :ourse in the sequence of supervisory train- 
ing courses. The evaluators would recommend that the test items be re- 
viewed wich respect to their potential fic to a more typical training 
population. It should further be pointed out that the te^t instrument 
was developed without having obs -ved the course in action. In the 
future it would be wise for evaluaLors to attend a course prior to 
attempting to develop either behavioral objectives or test items for 
that course. Since the train-e.., representing a rather high educational 
level group, failed to perform well on the application of principles 
part of the test, it may mean that insufficient opportunity was provided 
for them to practice these kind of behaviors required for the test, 
thereby requiring that the course deal more directly with this type of 
cognitive behavior if indeed this is the intended outcome. It*further 
needs to b-j kept in mind that the ultimate objective oi this series 
of courses is improved performance as a supervisor. At best, a paper 
aad pencil test can only approximate situations in which principles of 
supervision would be applied. It tells us nothing about how an 
individual woul actuall;/ behave in a given situation. In short ti-(Bn 
the most optimum way to evaluate a course of this type would be to put 
the trainees in a situation where they vjou 'i have to, by role playing 
at a minimum, display the types of behaviors that the course intends 
to transmit to them. In final summation, much more work is warranted 
on the evaluation of this course, and maybr ^ven course delivery in 
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order to make the evaluation of the course more congruent with its 
Lmmediate and ultimate objectives. 
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JOB PERFORMANCE EVALUATION 



The past three decades have seen a growing awareness of the need 
for training evaluation and the development of human performance measures 
that congruently and consistently reflect the outcomes of training. The 
massive training requirements of World War II and the subsequent explo- 
sion of technology with the production of man-machine systems of higher 
and higher complexity have produced an urgent need to study and measure 
human performance. This growing complexity in technology has produced 
a concomitant complexity in both training and performance evaluation. 
One might conclude, given the urgency of performance evaluation and 
present-day research and development resources, that problems of evaluating 
on-the-job performance would have been for the most part solved. Such 
is not the case. Most of the research in measuring human performance 
has been aimed at the development of criteria for the validation of 
selection variables and not for the evaluation of the efficacy of 
training. Wilson (1962) and Frederiksen (1962) in their thorough 
analyses of the problems involved in the measurement of human performance 
as it relates to training have recognized this deficiency in the area 
of training research. The training developer is confronted more and 
more with the pressing need for adequate criterion measures to evaluate 
outcomes of various instructional treatments in order to choose those 
which produce the best results. Frederiksen (1962) states: 

The homemade tests and course grades which used to be thought 
satisfactory are now seen to be inadequate. Criterion measures 
which more accurately reflect the objectives of instruction and 
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wl^'Lch permit judgment to be made separately about various aspects 
of the teaching program are needed . 

Wilson (1962) would even put performance evaluation before training 
development. He states: 

After all, one would reason, there is no way of knowing how to 
train people unless you can measure, first of ail, how they 
perform when they are finally assigned to r job. 
The amount of performance measurement research compared to the 
amount of training continues to be small. By far, most of this research 
has centered on the use of ratings and other gross measures. Only 
isolated attempts have been made to come to grips with real problems 
of measuring how well a man performs the actual task for which he has 
been trained. Beyond ratings, the vast majority of research reports on 
training assessment has embodied the more conventional approaches of 
classroom measuring instrumer^-s as the means of asscs:'^--^ training. The 
use of on-the-job performance measures has been rarely »'iCupted, and, 
except for the armed services, there has been little strong i.nterest 
in research and development in the area of practical performance 
measurement for trainir/g evaluation. 

There are two major reasons for the lag in development of per- 
formance methodology. First, this area of personnel research has been 
dominated by the quest for selection criteria. Development of rating 
scale methodology has provided measures of marginal, but sufficient, 
validity for the evaluation of selection variables. Second, the cost 
of performance testing is high. Performance tests are costly to build 
and costly to administer, requiring the measurement of one person (or 
few at most) at a time. Because of these costs training evaluators 
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have typically fallen back on the use of rating scales and written 
tests since these can be used with groups. From the perspective of 
developing effective training the trade-off involves validity per unit 
cost; that is, whether to develop and use highly valid but costly 
performance measures or inexpensive but questionably valid gross measures. 
The question remains unanswered in any general sense, but will depend 
on the constraints, value and need for precision of the particular 
training under consideration. 
Performance Measurement - State of the Art 
(Gene ral) 

The basic role of performance measurement is to provide feedback 
information on how well a man performs his job to those responsible 
for his training both in training courses and on the job. In concep- 
tualizing the performance measures as a kind modifying feedback 
information, in addition to the requirements of reliability and validity, 
Wilson (1962) derives three requirements from principles of human learning 
(McGeoch and Irion, 1952): 

1. The mf isures must be quite specific. In order to be effective 
they must identify specific aspects or elements of the job so that 
both trainer and trainee may take action to improve performance by 
improving these elements. 

2. For maximum utility the measures must be available to the 
trainer and trainee soon after the performance. 

3r Performance should be measured often, for by doing so the amount 
of reinforcement provided is increased with a consequent increase in 
the speed of learning toward the desired level of performance. 
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Because requirements two and three are problems for training and 
evaluation administration rather than characteristics of criterion 
measures, Wilson's study deals largely with the issue of specificity. 
State of the ^ \rt of Job Performance Criterion Measures 

Frederiksen (1962) discusses approaches to obtaining training 
evaluation measures that are also relevant to the art of job performance 
evaluation. He covers, for example, the advantages and disadvantages of 
soliciting opinions, administering attitude scales, measuring knowledge 
and eliciting behavior. The following list of the various approaches 
to job performance evaluation was synthesized from the works of Wilson 
(1962), Frederiksen (1962), Flanagan (1954) and Flanagan and Miller 
(1955 a, b, c, d). Each approach is accompanied by a brief descriptj-on 
of the current state of the art for that particular criterion meast-i^^ 
with special attention paid to the problems of specificity, nearness, 
to ''real world*' performance, cose and psychometrics . (A tabular treat- 
ment of the sam'=i state of the art is presented in Table 7.) 

1. Solicit Opinions. Job performance evaluation may be done by 
soliciting an opinion of the quality of the performance from either 
the worker's peers, his supervisor or an expert observer. The 
evaluator may solicit a descriptive opinion or an opinion structured 
by an instrument designed to measure the observer's impression or 
his observation of the trainee's behavior. The two instruments 
often used are rating and performance checklists, 
a. Rating scales are widely used because of their low cost, 
but they have little else to commend them. As defined by 
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Wilson (1962), "cumulative impressions or evaluations made 
by an observer and recorded at a time later than the 
observation . . • they are seldom designed with sufficient 
specificity to provide feedback to training design. Taylor 
(1958) suggests that the opportunities for observation seldom 
occur and that observers often lack the skill for effective 
observation. Thus rating scales are at best measures of 
impression. Because of their low cost, they are widely used 
nevertheless. In such cases they can be made rrore specific, 
objective and realistic if carefully referenced to the tasks 
performed on the job by trainees. 

b. Performance checklists are often more structured as 
criterion measures than are rating scales allowing the 
observer to attend to specific task elements and providing 
an externalizable scale for direct observation. 
2. Administer Attitude Scale. If the objectives of a train:" 
program include changes in the trainees' attitudes, the evaluator 
may simply ask trainees to what extent they have adopted the desired 
attitudes, a mp.thod that includes all the drawbacks of soliciting 
opinions. A more sophisticated approach is to administer an 
attitude scale on which all items "are .elevant to a single factor 
and contribute to a single score . . . ( Frederiksen , 1962). 
The method is easily administered, hence relatively inexpei»f!ive , 
but is susceptible to bias due to the trainee's response set. The 
bias can be corrected to a degree by the "^forced choice technique." 
For data of sufficient specificity fr^r feedback to training, 
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attitude scales should measure the trainee's attitudes in terms of 
changes in his behavior, but there is little evidence that scores 
on attitude scales are correlated with behavior to any marked 
degree. ( Frederiksen , 1962). 

3. Measure Knowledge. Although the art of evaluation is highly 
advanced at assessing the results of knowledge training, program 
objectives that anticipate a transfer of training to on-the-job 
performance must be stated in terms of behavior. Knowledge 
tests assess knowledge, not behavior or task performance; they do 
not provide the type of data that can be used to assess the effi- 
cacy of any training other than knowledge training. (Frederiksen, 
1962). 

4. Elicit Behavior. Because behavioral objectives and outcomes 
are prerequisites of effective training, evaluation also should 
be based on the observation of behavior as it occurs in the "real 
world" or as it can be elicited from trainees for the specific 
purpose of evaluation. Because of psychometric problems involved 
in observing "real" behavior, cited by Taylor (1958) and mentioned 
above, the evaluator can elicit behavior from trainees that can 
be structured in the following ways: 

a. Eliciting Related Behavior. Bases evaluation of performance 
on the observation of behavior that is logically related to 
the "real behavior that is in some way difficult to observe. 
Frederiksen illustrates the method with the evaluation 
of writing performance ("real" behavior) based 



104 



98 



on rewriting or editing exercises (related behavior). 
The evaluator is attempting to measure behavior but can 
not be sure of the' validity of his criterion measure, 
which is related to ''real'* behavior solely on the basis 
of logic, 

b. Eliciting What I Would Do" Behavior, Presents 
problem situations for which the trainee provides 
solutions; thus the evaluator can observe behavior of the 
type the trainee performs in the "real world," but, as 
Frederiksen- suggests, the only accurate answer to 

such items is "It depends," If the instrument presents 
a multiple choice the trainee chooses solutions rather 
than invents them; in any case he probably chooses a 
"correct answer," not what he would do if performing on 
the job. 

c. Eliciting Lifelike Behavior. Attempts to observe the 
examinee's behavior performance as if he. were performing in 
a "real life" situation. Implementation of the method 
varies from pen and pencil to situational items. It 

is preferable to knowledge tests, attitude scales and 
opinion solicitation because it ia based on the observation 
of behavior, in fact it more nearly approximates on the job 
performance than the methods of eliciting behavior men- 
tioned above. The only method nearer to "real life" 
behavior in which psychometric problems are insurmountable. 
In short, given the current state of the art, eliciting 
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life like behavior is the best method of evaluating the 
transfer of training to job parformance. Frederiksen (1962) 
calls it the method "recommended for first consideration as 
a measurement technique," Wilson (1962) describes work 
samples, which are one way of eliciting life like behavior, 
"very valuable aids in the measurement of job performance." 
7. Observing "Real Life" Behavior, This is the most "real" of 
criterion measures, but it is also beyond standardization. The 
occurrences of the behavior for which measurement is sought may 
be rare, making the method very costly in time and effort; when 
the behavior occurs it can not be standardized. The methodology 
* provided by Flanagan (1954) and Flanagan and Miller (1955, a, b, 
c, and d) has provisions for observing, recording and analyzing 
"real life" behavior. While this is short of standardization and 
measurement, it nevertheless provides a powerful tool with which one 
can construct variables that can be valid and consistent measures 
of job performance. 

In sum, the evaluator should observe behavior that is as nearly 
"real" as possible in order to obtain data of a degree of specificity 
sufficient to provide feedback to training design. The state of the 
art of job performance evaluation makes eliciting "real life" behavior 
the best method for measuring the transfer of training to job performance. 

Perhaps the best method of eliciting life like behavior is the 
performance test. Four exhibits (A through D) of performance tests, 
reproduced from a manual of performance tests prepared for the U, S. 
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Navy (Mackie, £t al^, 1953), are presented as good examples of 
instrumentation and measurement of specific task performance. Each 
perTormance test exhibit Includes a description of the test, the scoring 
method, evaluation of the test, statistical results, and a specimen of 
the test itself. 

Exhibit A, Circuit Trouble-shooting, and Exhibit B, Ground 
Testiiig, are performance tests for Navy electricians' mates. Both A and 
B use actual equipment and instruments the men trained would use on 
the job. 

Bor.h elicit troubleshooting behavior from the trainees. One 
interesting aspect of the performance tests in A and B is that the 
conditions of performing are under the examiner's control; that is the 
examiner can manipulate both the difficulty and kind of performance 
required. The performance itself is under the trainee's control and 
is explicitly observed and scored. The conditions of observing and 
testing permit not only objective and explicit measurement, but immediate 
diagnosis and feedback to the trainee. Thus, performance tests become 
powerful learning tools. 

Exhibit C, Use of Common Hand Tools, is not strictly a performance 
test in the same sense that A and B are. However, it is much more than 
a simple tool naming test as it requires the actual selection of the 
appropriate tool to perform specific tasks- Thus, the trainee must 
understand the job task he is faced with and relate the correct tool to 
it by name. The test is practical being directly related to job perfor- 
mance requirements and affords the opportunity for direct learning by the 
trainee • 
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Exhibit D, Transmitter Tuning, is what can be called a procedural 
check-lis'c performance test a^.d illustrates another approach to 
performance testing. Like the previous performance tests the test shown 
in D utilizes the actual equipment and all the tasks demanded on the 
job. The performance of the task required by the test in D is sequential, 
each step being critically dependen*: on the one before it. If the trainee 
makes a mistake the examiner gives him feedback by prompting him. Thus, 
the examiner insures that each step is finally performed accurately. 
The trainee is scored objectively on the number of correct steps 
executed in the task performance and can be diagnosed on the spot for 
any special difficulties he is having. 
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EXHIBIT A 
CIRCUIT TRCUBLE-SIICCTING 

Dp5;cription 

The equipnent for this test consisted of a liglitinG circuit mounted on 
a bulkhead. VJliile the cables were shorter than would normally be found 
aboard ship, tlie circuit units and wiring were essentially tiic same as 
standard shipboard circuits. Two opens, a ground, a short, and a blown fuse 
were put into tlie circuit. 

Using hand tools, a voltage tester, and a megger, the examinees were 
required to locate the casualties in tlie circuit. They recorded their 
performance by placing an X at the 3ocation of the casualty on the circuit 
diagram (see recording form, page 25) and by v/riting in the name of the 
casualty, (short, open, ground, or bad fuse). 

Scoring 

Two points ivere given for eacli casualty correctly located and named. 
One point ivas given for indicating a branch of the circuit tliat contained a 
casualty (without locating it precisely witiiin tliat branch) if tlie casualty 
were correctly named. The total possible score was 10. 

It was necessary to score the tests in this way because of tlie ex- 
aminees' metliods of working. Beginning with tlie 10 wire connection box, 
tliey first determined which brandies of the circuit contained casualties, 
and then traced them down, isolating tliem witliln tlie branches. Frequently 
they could determine uiiich branch contained a casualty but could not 
determine its exact location within that branch. 
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Rel i ahi 1 ir .y 

Alternate forras of ti:e test were constructed and administered to 60 
examinees. Form C ivas administered four days before tlic administration of 
Form A. Tiie distribution of scores on Form B was Uicjlily positively skeived 
witii a mean score of 2.9 and a median score of 1.9. The distribution of 
scores on Form A more nearly approaclied normality ivitl! a mean score of 4.5 
and a median score of 4.3. 

TI]e correlation between scores on the two form-, ivas -50, whicii is 
reC'^^rded as a conservative estimate of the reliability, because of the 
foUowinfj factors which may have contributed to the unreliability: 

1. The tests were administered to the trainees in tlie last v/eek of the 
Ei.I School course after they had completed the sclioors final 
written achievement and final performance examinations. The ex- 
aminees felt tJiat their performance on this .st ivould not affect 
their school standing, and consequently, motivation was low. 

2. Changes in the nature of the casualties from one test adminis- 
tration to the next and from one set of equipment to anotlier. 
Although the Navy personnel assisting in tlie administration of the 
tests were interested in testin^j and conscientious in their worl:, 
they oft..i could not see the necessity for making the casualties 
identical from one test administration to the next and from one set 
of equipment to another. (There ivere five identical sets of equip- 
ment for this test.) As a result, project personnel could not 
always be sure tliat the examinees were working on comparable sets 
of equipment. 
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3. Changes in the casualties as a result of the ivorl: of tl:e excimijiees 
It v/as observed that additional casualties occurred in the cir- 
cuits as a result of errors by the examinees. The circuits were 
energized, and it was possible for the examinees to burn out a 
fuse while they were working. TJiis problem was corrected after 
the test Jiad been administered a few times. 

Eval unti o n 

A large percentage of the Electrician's Mates* work aboard sliip con- 
sists of trouble-shooting electrical circuits. Tliis test demands the same 
type of I'erfornance that is required in the practical situation. Conse- 
quently the test served not only as a measure of achievement but also as 
an aid to training. 

The test was at first a difficult one for the EM Scliool trainee?. 
The distribution of scores on Form A was highly positively skewed. Cvor 
a period of four classes, during which time mean scores increased from 
2.2 to 3.C. the distribution became more syr/imetrical . 

In the interests of simplicity and economy, tlie diagram of the cir- 
cuit the examinees were to trouble-shoot was also used as the recording 
form. This approach to recording performance may have served to reduce 
the reliability of the test, since it was difficult in some cases to deter- 
mine exactly where an examinee tliouglit the casualties were located in terms 
of the ]C s he marked on tlie diagram. It might have proved better, for 
example, to list tlie units of the circuit in a column at the left of tlie 
page and the possible casualties across the top. The examinees could tlien 
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have recorded tlieir performance by placing X's in the appropriate cells. 

Scores on tlie Circuit Trouble-Shooting test correlated with scores on 
other tests in the battery from .15 to .44, the highest being with Controller 
Trouble-Shooting scores. The scores correlated .35 with final class standings 
(N = 159). The correlations with General Classification and L'echari.cal 
Aptitude test scores were .04 and .15, respectively, (N - 140). 
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TEST Rr..siir.T.<^ 



CIRCUIT TRCUDLZ-SIICCTirvC 



FREQUENCY 




RAU sccr:. RE.'^IJT.TS 

L'ean = 4.2 
riedian = 4.0 
S.D. = 2.4 

Alternate Form: .50 
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Ti-JT ^^^!HC 105 B2 
SEPTET'BER 1952 
NAME r 



DC:_ 
MHC: 



RATE: 



PLUS SCORE: 
UimS SCORE: 
CLASS: 



CIRCUIT TROUBLE SHCOTING 

Dirjctions ; THE CIRCUIT 15 Er.'ERGICED. EELO'.Y IS A DIAGPu;>;; OF THE CIRCUIT. 

:HFK GIVE!: THE V.ORD BEGIN' , YCU wTLL LCCATE AS r.;A;IY SHORTS, OPENS, 

OR BAD Fuses iw th:-: :ip.guit as you ca:.i. if the circuit is grounded,, locate the 

3GuF.CS OR SOURCES 0? THE GRCUxVD. INDICATE ON THE DL4GRA.II V.HAT FAULTS YOU FI.\T! 
AND -HEPE THET ARE. YOU l^AY DISCONi-.'EGT A^.T OF THE V,1PE3 OR REMOVE THE FUSES. 
DO NOT CCRPECT ANT CF THE F.-VoLTS YOU FINTD. AFTER Tj;:E 13 CALLED YOU V/IU. 
REASSJI.iPLE THE EiUlP;.;ENT AND LEAVE IT JUST AS YOU FOUND IT. 

DO THE V.-ORK JUST AS IF YOU VERE /.ECARD SHIP. OBSERVE ALL SAFETY 
PRECAUTIONS nND ST/A.DAi-lD PROCEDURES. APJi TiiSRE ANY CUESTIONS? 

Tir,G Linitj T'.TENTY MINUTES 



S & R BOX 



_ _FU_SED_SVm'CH. PANEL 
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GROUND TESTING 

nescription 

The examinees were required to take ground readings on eight different 
pieces of equipment commonly used by Electrician's Mates: 

1. Dead Front Switch Panel > This switch panel has all of the "live"' 
parts enclosed in heavy sheet metal with only the switch handles 
exposed. It is better known as the Main Switchboard aboard 
ship* 

2. Compound D.C> Generator 

3. Live Front Switch Panel s This switch panel serves the same 
purpose as the dead front switch panel, but differs in that 
the switches are exposed. It is used as an auxiliary switch- 
board aboard ship. 

4. Shipboard Lighting Circuit Panel 

5. Power Panel 

6. Three Phase A.C. Motor 

7. Controller . This was connected with the A.C. motor. 

8. Fuse Panel 

Grounds were placed in some of these equipments, and others were left in 
a normal condition. In order to obtain correct ground readings, the examinees 
had to perform the correct preliminary operations, for example, open or close 
switches, lift the brushes from the generator to isolate the field from the 
armature, remove fuses, etc. They also had to be able to use common hand 
tools and operate a megger. 

Scori ng 

One point was given for each correct ground reading. The total possible 
score was 33. ^ a ^ 
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Readings taken on the Dead Front Switchboard wrre not counter! in the total 
possible score, TJiis was a brief warm-up period, utilized as such bccuaso the 
resistances to ground in this equipment could not be controlled from one ex- 
aminee to the next, 

Rfiliahi 1 i ^ly 

Tivo estimates of tno internal consistency reliability of this test were made, 
The first estimate was based on s'^ores on odd and even units, for example, odd 
and even switches, counting both poles of a switch (positive and negative) as a 
single item. The corrected correlation between odd and even iteT^ scores using 
this method was .06 (N = 100). 

The second estimate was based on odd and even responses vjilhir^ vn>ts. For 
example, the negative pole of switch number 1 and t;;o positive poL': of switch 
number 2 were counted as odd items. The other poles of these switches were 
counted as even items. Using this method, the corrected correlation between odd 
and even item scores was .94 (N - 100). 

The Ground Test called for the examinees to work with a broad sample of 
electrical equipment found aboard ship. It demanded a knowledge of the inter- 
relationships among these equipments and the skill to isolate them from one 
another for purposes of taking ground readings. Results of a test of this 
type should give a good indication of Electrician's Uates' familiarity with 
electrical equipment, and their ability to work with it. 

An evaluation of the test results at the El\ School clearly indicated that 
some of the trainees not only did not know where readings should be taken on 
the equipment, but that some of them actually did not know how to use a megger, 
a simple, but indispensable device to Electrician's Hates. This discovery 
"sold" some of the previously skeptical Chief Petty Officers on the 
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value of performance tests. 

The distribution of scores shows a range from 3 to 31 with good discrim- 
ination among the examinees. 

The correlation between raw scores and the time taken to perform the ta.sh 
was -.02, N = 105, again indicating no relationship between speed and adequacj 
of performance. 

The correlation between Ground Testing scores and other test scores in 
the battery ranged from .06 to .26. The correlation with final class stand- 
ings was .22 (N = 108). With final written achievement examination scores, 
the correlation was .21 (N = 108). Ground test scores correlated .11 and .08 
(N = 97) with General Classification and Mec'ianical Aptitude test scores, 
respectively. 
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TS ST RCSIILTS 
GROUND TESTING 

30 




0 3 6 9 12 15 10 21 24 27 30 

SCCRS3 



RAU sccRE RSSL1.T5; Tif.n: 

i.Iean - 19.4 f.Iean - 37.2 (Minutes) 

f.iedian - 19.5 S.D. - 13.3 (Minutes) 

S.D. - 5.9 

RELI/.DILITY 

Cdd-Even Grouped'-' Responses: .06 (Corrected) 

Odd-Even Single'-' Responses: .94 (Corrected) 
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TEST ISIRC 105 El 
S2?Ta:ESR 1952 
N/J.IE: 



START : 



FINISH: 



Ri\TE ! 



liyW; SCORB! 
TI1.IE: 
CLASS: 



Directions : 

TERIvIINAL 13 
A IS ON THE 
TUffi, EXTRA 
PRECAUTIONS 



GROUND TESTING 

TAKE GROUND READINGS ON THE EQUIPMENT LISTED BELOV/. RECORD YOUR 
READINGS IN THE BLANK SPACES PROVIDED. NOTICE THE NEGATIVE 
ON THE LEFT ON BOTH THE ECUIPI/IENT AND THE ANSFER SHEW. ALSO, 
LEFT ON EOTH THE EQUIPIaENT AND THE ANSWER SHEET. DON'T WASTE 
CREDIT v/ILL BE GIVEN FOR RAPID ..'ORK, BUT OBSERVE ALL SAFETY 
. ARE THERE ANY QUESTIONS? 



ERIC 



1. DEAD FRONT S'./ITCH BOARD 



Switch No. 1 



Ne (ya t ive ( Pos i t ive 



Sv/itch No. 2i 



Switch No. 3 



Sv^itch No. A 



2. G5N5R;.T0R 
Armature : 
Field: 



3. LIVE FRCOT 3V/TTCH BOARD 



Negative Positive 



Switch No. 1 



Svo.tch No. 2 



A. LIGHTING Tkl^L 



[Negative Pes iti ve 



Sv/itch No. 1 



Switch No. 2 



Switch No> 3 



Switch No. 4. 



5. POVffiR PANEL 



Switch No. 1 



Switch No. 2 



Sv/itch No» 3 



6. A. C. MOTOR 



7. CONTROLLER: PILOT CIRCUIT 



8. FUSE BOX 
Fuse A 



Fuse A 



Fuse B I Fuse C 



Fuse B 



Fuse C 



KEEP YOUR ANS.TEH SHfilL-T FACE POV/N 
wHEM NOT l/IUTING 
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USE OF COMMON HAND TOOLS 

Description 

This test was not, strictly speaking, a practical performance test, bu: 
probably resembled a performance test more than a written test. The ex- 
aminee sat before a tool board coTitaining? 66 different tools, each of which 
was numbered. He was required to indicate the tool or tools he would use 
to do a task which was described on the test answer form. Twenty-two dif- 
ferent tasks were described, and the subject responded by writing the number 
of selected tool(s) on his answer sheet. 

The time limit on the test was twenty minutes, which was sufficient for 
all examinees to finish. 

Scoring 

One point was given for each properly selected tool. The total possible 
score was 29. 

Reliability 

The corrected internal consistency reliability of the test as estimated 
from odd-even item scores was .Tl, 

Kvaluation 

This test provided a measure of men's knowledge of the uses of various 
hand tools. While it cannot be stated that the test scores gave an indi- 
cation of how men actually use the tools in performing practical tasks, it 
is assumed that knowledge of which tool to use is necessary before the tools 
can be employed properly, 

A test task might be devised which required examinees to demonstrate the 
proper use of a wide variety of tools. Such a test probably would be highly 
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time consuming, however, rind require an examiner for each examinee. 

The distribution of scores approached normality with a mean score of 20.2 
and a median score of 20.6. The test discriminated well among the examinees. 

Tool test scores correlated with other performance test scores from .04 
to .27. Correlations with the final written achievement examination and 
final class standings were .20 and .36, respectively, (N = 245). The latter 
correlation is slightly spurious, since tool test scores in part determined 
the final class standings. The correlations with GCT and MAT scores were 
.03 and .23, respectively, (N = 228). 
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TEST nn:sui/fs 

USE CF CCiiT.iCN FJAND TCCLS 
50 




scenes 
nxj sccnz cssilt." 

/.lean =-20.2 
liedian =20,6 
S.D. = 3.9 
nniLIABILITY 
Cdd-Even Items: .71 (Corrected) 

i 3 .V 

ERIC 



EXHIBIT C 



116 



ERIC 



SEPTi-'BER 1952 

TioT f.ftlRC - 105 ^1 

J, SCORE: USE SCORE: TOTAL SCOPE: OR>\DE: _ 

RATE: CUSS:, 

N.".r^E : , — 



USE OF C0:210N HA^ TOOLS 

nirect^ons- IN TK:i BLANK APPEARING AFTER EACH OF THE JOB QUESTIONS BELOU, 
W^^^^^O.-^^^ TOOLS 2V~,^^ T^^^^ 

UNNECESSARY TOOLS. HERE IS A 3Ah;PLE TEST QUi^oTlON: 

A. V;hat tool or tools would you use to remove a round r.^tal 

pin from an armature shaft? ^ ' 

NAMES? Pin or drift punch, ball-p een harrjner 

.00.3 .0. ■j^'^^.cii^^i^^^^Pigr 

APJi; THERE ANY QUESTIONS? 
Time Limit j TVfENTY MINUTES 



1. vmat tool or tools would you use to compare the diameters 
of two wires? 



NAMES? 



2. .That .ool or tools would you use to loosen the nut shown 
belov/? 



N/diiS? 




Y,'r.at tool or tools would you use to cut off a section of 
copper tubing? 



NiUIES? 



^ If ynu did not have a stop v:atch, v/hat tool or tools would 
you use to determine the speed of a rotating shaft. 

n;j.iE3 ? . - — 
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Hero are t;vo views of a set scrcv;. V/hich tool or tools 
v;ould you use to tighten this screw? 

NAliSS ? 

Top View Side View 

I I I i 

■,nat tool or tools would you use to make round hole:^ 
in a gasket? 

N/vi-ES? 



V.liich tool or t,oo1s v/ould you use to measure the nmber of 
threads per inch on a 1//+" diameter bolt? 

, NAI.;3S? 



\."hich tool or tools would you use to loosen the r.acnine 
screv/ pictured belov;? 

NhJ.lES? 



Top View 




V.'^iich tool or tools v;ould you use to measure the depth 
of a 3/8'' diameter hole? 

NA!.iE3? 



l.Tiich tool or Lc'^ls v/ould you select to measure the 
dianeter of a roc (approximately 1/4" in diar-.eter) to the 
nearest one-thousand oh of an inch? 



..r.at tool or to'jls v;ould you use to r.easure the dictance 
between controller contact points? 

H/j.lES? 



vrnat tool or tools would you use to keep a shaft fron 
turning while the end nut was being tightened? 

Ni\I^!ES? 
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i3, Vrliat tool or tools would you select to count the number 

of turns a shaft nakes? 

N.Hl;lES? 

'./hat tool or tools v^ould you use to cut and dress a piece 

of 3/32" (thick) sheet copper? ^ 

nai.:es? ^ 

15. .That tool or tools would you use to cut off the head of 

a bolt rusted to a steel plate? 

N/\l.Ii3? ^ ^ . 

. 16. List all the tools you would use to tap full threads to 

the bottom of a blind hole? 

NAJ.1ES? ^ , 

17, In addition to a straight edge, wh^t tool or tools would 

you use to draw lay-out lines on a piece of sheet copper? 

NAI.ffiS? 

18, V.^at tool or tools would you use to remove the nuts hold- 
ing the lugs in a 20 wire connection box? 

NAI.IES? , 

19. V.Tiat tool or tools v/ould you use for making a l/4" round 

loop in the end of a piece of wire? 

NAlffiS? . 

20, '..Tiat tool or tools v/ould you use to loosen a 3A" nut on 

a battery? 

NA!,:ES? , 

21. List all the tools you v/ould use to nake a lA" hole in a 

l/2»' thick steel plate? 

N;u/J£3? 

22. '..hat tool or tools v/ould you use to make a 1/A - 20 stud 

from 1/4" round stock? 

nj.:es? 
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TRANSiMITTSR TUNirJG 

Doscrintion 

This was a procedural check list type of performance test. The examinee 
was required to time the TDE .transmitter, which was selected because of its 
wide use and because it embodies -all the basic points of transmitter tuning. 
As the examinee performed the tuning task, an examiner observed. and re- 
corde<l whetlier or not he performed .each step correctly. . 

Scoring 

Scoring this test presented some interesting problems. Since per- 
foi-mance on each successive step .in the test task was to some extent de- 
pendent on performance at prior steps, an examinee might make a whole ' 
series of incorrect adjustments simply because he was "off" on some early 
step in the procedure. Furthermore, Jiis subsequent adjustments might have- 
been perfectly appropriate, assuming » as the examinee, would, that the early 
step vas correct. 

The problem, then,.. was to score the examinee's performance in some 
fashion which would properly reflect the number of steps done correctly 
and avoid this correlation of errors. The solution \ms to prompt the 
examinee «acli time he jnade an error or was "off". Thus each examinee 
either did a given step correctly by himself or was prompted so that in any 
case the step was properly completed. This assured that. each examinee 
performed a given step with the equipment in the same condition, thus help- 
ing to standardize the task. 

The examinees score was the number of steps completed tvithout prompting. 
The total possible score was 42, 
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Since experienced men consider that time taken reflects skill in the 
transmitter tuning task, time was also used as a score. The correlation 
(Spearman rank-difference, U =. 40) between time taken and raw scores was 
found to be .20. Thus, at the trainee level, at least, there was little 
relationship between liow well a man tuned the transmitter and how long he 
took to do it. 

Some of the items on the test required rather complex responses while 
others required only the turning of a switch to one of two alternative 
positions. Tor this reason, a system of weighted scores was tried. 
Differential weights were obtained by averaging the judgment of several 
experienced men. The rank order correlation between weighted and un- 
weighted scores was found to be .95, (N - 71). Scors based on weighted 
items produced .practically the same rank order of examinees, but did 
serve to increase the dispersion of scores. 

Reliability 

The test was administered twice to 40 men with a time interval of two 
weeks between administrations. The test-retest reliability computed from 
these data was found to be .09. (unweighted scores). 

It is of some interest to note that, in retesting the forty men. the 
assignments of equipment, examiners and frequencies were different from 
those made during the first administration of the test. This indicates 
that reliability of the test was not a function Cx particular examiners or 
of practice on a particular transmitter at a particular frequency. 
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Evaluation 

The test discriminated among striker RJ.I's or tlie basis of botli time and 
raw scores. The raw scores ranged from 17 to 42 with a mean score of 34.1 
and a mediar score of 34.6. The time scores ranged from 4 to 10 minutes 
with a mean of 10.4 and a median of 10.. 1 minutes. 

Transmitter tuning scores and other RU performance test scores cor- 
related from .00 to .39. The highest correlation v/as with Circuit Drill 
scores. 

l/hile the test required one examiner for each examinee, its discrimina- 
tion ability and reliability warrant its use as a performance measuring 
device. In addition the test should prove to be an excellent training 
procedure both afloat and ashore. 
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TEST RI-SULTS 



TRANSLIITTER TUNING 




40 42 
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RA!J SCCnS RESULTS 

riean = 34.1 
iledian = 34.6 
S.D. = 5.4 

urn. 

fiean = 10.4 
S.D. = 3.2 
RELIASILITY 
Test-Retest: C9 
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t;-:3t ;;::f?c 105 - TTCb 
DECii:nji;R 1952 



INSTRUCTION SHEET 
TRANSMITTER TUNING TDE (HF) 



Instructions for examiner ? 

1. Use frecuency above 3.0 mcs. The same frequency is to be used for all examinee 

2. Have the frequency meter already set to the desired frequency. 

3. Set all transmitter controls to positions that v/ill require re-setting by the 
subject being examined. 

Examinees are to be tested one at a time. Do not allow any examinee to watch 
another bein.e test:='d. 

5. Directions for scoring each examinee: 

NOTE: For each person bein,? examined there will be two scores obtained by the 
examiner J 

a) Time. The examiner is to time each person for Parts I through V with a 

stop watch and enter this time in the blank provided in the right 
hand corner of each record sheet, 

b) The number of prompts . The examiner is to observe the tuning of the 

transmitter closely and proript the examinee, if 
necessary j on each of the steps of the tuning procedure. Scoring 
of each step will be as follows: 

If it is necessary bo prompt the subject because he makes a 
mistake in the procedure (v/rong dial setting, wrong meter read- 
ing, etc.) or he forgets the step, incircle the zero (O) before 
each iteraj 

1 (a) 

If the subject is able to perform each step correctly with no 
errors and requires no prompting, encircle the number one (l) 
before each item: 

® 0 . 

&• Read the following instructions to the examinee: 

"You are to be tested one at a tine. You will not be allov^ed to watch any other 
person beincr tested. You are to tune this transmitter all the way through for 

operation on this frequency: KC5* This probably v;ill involve 

the use of all controls A through K. The frequency meter is already set up for 
you,»» 

"You will be observed and prompted if necessary on e^ch step. *7ork as rapidly 
as possible. Are there any questions?" 

O Tell the examinee to bep^in and start the "stop watch, 
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TIME:_ 

JCOREt 



RECORD SHEET 

tr;.ns:-!IttE'^. TUiinio tde (hf) 

DESIRED FRECUSNCY_ ^KCS 

START 

PART I FHELIinNARY SETTINGS 

1 1. Set the following' controls and switches in the indicated positions! 

CONTROL POSITION 

1 0 HF-IF switch, control A HF 

1 0 Remote-Local switch Local 

1 0 Ad just-Tune-Operate Adjust 

1 0 CVM.'iCu-Voice switch CW 

1 0 Antenna couplin? control K 15-20 

1 0 Antenna tuning capacitor control M 50 

2. Set the follovdn? controls to the position of the approximate frequency 
to be set up: 

COk'TROL 

1 0 B 
IOC 
1 0 B 
1 0 E 
1 0 F 
1 0 G 
1 0 H 
1 0 J 
ION 

PART II : 'ASTER OSCILUTOR TUNING 
1 0 3. Press the start but.+— ^.id adjust filament voltape 10 volts, 
10 4. Adjust pldte voltage to 2000 volts. 

10 5. Press test key on handrail and tune master oscillator, control C for ze 
beat in headphones. 

PART III RESONATING ALL STAGES 
(Test key must be pressed to tune all following stafres) 

Shift Ad just-Tune-Operate switch to tune position. 

Adjust control E for maximum I. A. grid current. 

Adjust I. A. tuning .ontrol G for maximum P. A. grid current. 

O . : 0 9. Adjust P.A. tuninp control J for minimum F.A. plate current. | 

^ilMl^i^ 1 



1 


0 


6. 


1 


0 


7. 


1 


0 


8. 




0 


9. 
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1 0 10. Adjust antenna inductance for ncixijnun P. A. plate current. 

(If no peak is obtained, does he shift ant. feed sw. to other position?) 

1 0 11, Adjust antenna capacitance for maximum F.A. plate current. 

1 0 12* Re-adjust P. A. tuning control J for minimum P. A, plate current. 

PART IV ANTENNA COUPLING j TUNE 
1 0 13. Increase antenna coupling for 70 ma. of P. A, plate current, 
1 0 lA. Re-adjust P. A. tuning control' J for minimum P. A. plate current. ' 
1 0 15. Re-adjust antenna inductance for maximum P. A. plate current. 
1 0 16« Re-adjust antenna capacitance for maximum P. A. plate current. 

PART V ANTENNA COUPLING: OPERATE 

1 0 17. Shift Ad just-Tune-Operate switch to operate position. 

10 18. Re-adjust P. A. tuning control J for minimum ?,A. plate current. 

1 0 19. Re-adjust antenna inductance for maximum P. A. plate current, 

1 0 20, Re-adjust antenna capacitance for maximum F,A. plate current. 

1 0 21. Is plate tuning at a minimum with plate current at approximately 

100 ma? 

STOP 

PART VI FINAL CHECK BY EXAillNSR 

Directions ; The examiner is to check the transmitter itself and ansv/er 
each of the follov^ine items. Encircle yes or no. 

YES NO 1. Are filament and plate voltages correct? 

YES NO 2. Are any of the current meter readings too high? 

YES NO 3. Is each stage tuned to the correct frequency? 

YES NO Are grid current meters peaked exactly? 

YES NO 5. Is P. A. plate current at exact dip? 

YES NO 6. Are antenna inductance and capacitance peaked? 

YES NO 7, Is it overcoupled? 

YES NO 8, Did he depress the test button only while tuning? 

13:^ 
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State of the Art - TDCA 
The state of the evaluation art for TDCA training nakes eliciting 
lifelike behavior with a performance test an expecially desirable 
method for measuring job performance. The scope of TDCA ' s training 
ranges from maintenance training to training in the principles of 
supervision. Such diversity must be met with a diversity of measurement 
methods and, more specifically, with a diversity of performance tests. 
While the cost of performance tests are relatively high when compared to 
less effective evaluation methods, the cost should not be seen as 
solely a cost of evaluation. Performance tests, when properly cons tiructed , 
can be used as highly effective training devices and as end-of- training 
evaluation instruments. The use of the same instrument for three 
purposes will not only spread its cost but will also provide a degree 
of consistency between three components of DOES- 
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COMMJNITY IMPACT OBSERVATION 

A General Description of the Problem 
The decade of the sixties, largely influenced by President Johnson's 
Great Society program, witnessed enormous outpouring of federal government 
resources toward the alleviation of social and educational problems. Many 
of the programs initiated during this era were conceived and operationalized 
literally overnight and funded liberally. One of the assumptions made 
was that government and the professions at all levels were capable of 
implementing progr ^ addressed to priority needs that would produce 
favorable results in a short period of time. Following upon the heels of 
this movement, was an increased concern for evaluation and accountability 
with respect to the social and educational programs. Government officials 
at all levels began to inquire about the impact of various programs 
upon target populations. At this time then from a national 
perspective great emphasis was placed upon evaluation of program. 

The Texas Department of Commianity Affairs, a state agency heavily 
involved in community improvement programs, also began to inquire about 
the effectiveness of its programs upon clientele in Texas, Tn particular, 
the executive director expressed a desire to know if funds under Title VIII 
of the Housing and Urban Development Authority, dealing directly with 
training of local government officials, were producing any effects. In this 
regard the question was phrased as follows: "Are our training programs 
having any impact on local community development?" Impact was defined by 
TDCA as the ability of local government to respond to local needs. Thus 
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we have the general origin of the need for evaluation that resulted in 
the development of DOCS. 

This report vill center on various aspects of community impact 
evaluation, and present a brief treatment of the historical background 
regarding movements that affect development of "community impact" 
evaluation. Then the specific conceptual and methodological problems 
will be discussed; recommendations will be presented to the Texas 
Department of Community Affairs with respect to this type of evaluation. 

Historical Background 

In dealing with community impact evaluation one is attempting 
to assess the quality of life in a community either directly or indirectly. 
Attempts to deal with this issue can be traced back to the presiential 
administration of Herbert Hoover (Wilcox, 1972). In 1929 a commission created 
by Hoover was to conduct a survey of social trends within the United States. 
The commission's report pointed out the complexity of social trend analysis 
in attempting to understand social phenomena. The commission recommended 
the convening of interdisciplinary panels for the purposes of developing 
what are now called social indicators. The onset of the depression of 
the thirties clearly sidetracked this work. The emphasis shifted to the 
development of economic indicators to monitor the recovery of the economy 
of the United States. 

Due to World War II and other pressing national concerns, interest 
in social indicators appeared to wane. During the fifties and the sixties 
the National Aeronautics and Space Administration became concerned with the 
possible long range consequences of space exploration with respect to 
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ways In which it may impact upon the social life in the nnited Stares. 
Out of this concern a commission was convened to explore the social 
ramifications of the NASA program. This effort produced the oft-cited 
book. Social Indicators by Raymond Bauer (Bauer, 1966). From this time on 
there has been a growing interest in monitoring social conditions within the 
United States and a considerable body of literature is beginning to 
emerge dealing with this problem. 

Another significant event occurred in 1966 when President Johnson 
commissioned the Department of Health, Education and Welfare to develop 
sets of statistics and indicators of general societal development that 
would be analogous to the broadly used economic indicators. In 1969, 
the commission's publication Toward a Social Report was circulated. This 
report attempted to deal with the complexity of issues, the definition 
of concepts and proposed methodologies to deal with social accounting. 

Since 1969 there have been many attempts to isolate and clarify 
issues and to define further key concepts within the field of social 
indicators. However, the field is still beset with many theoretical and 
conceptual problems in addition to research and evaluation methodological 
problems. In short, while progress has been made, the general field is 
far from having the conceptual and the operational rools to perform an 
adequate job of monitoring and evaluating improvements in the quality of 
life as a function of social programs. 

Emerging Evaluation Methodology 

Another important movement during the sixties was taking place 
within the field of education. This movement toward a reconceptualization 
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uf the purposes nnd the methodologies for eval.u;iting the outcomes of 
educational programs is of great significance to DOES. Prior to the 
mid-sixties, educational evaluation had been concerned primarily with 
outcome evaluation, that is making judgements about the effectiveness of 
programs based on learner performance at the end of a given learning 
sequence. To a large extent this orientation toward evaluation was 
influenced by the standardized educational testing movement. In turn 
testing movement was influenced from the 1920 's through the mid-1940's by 
the needs of the armed forces. This movement toward standardized testing w 
largely influenced by the emerging discipline of psychome tries . Psycho- 
metric theory, the concern for norm- reference testing and methodological 
issues related to reliability and validity captured the essence of the 
evaluation movement until the mid-sixties. 

Lee Cronbach (1964) gave impetus to the reconceptualization 
of the purposes of evaluation. He questioned"^ the purpose of evaluation 
centered on end-of-course achievement and proposed that evaluation 
directed tov;ard the improveir'^^ut of courses whi^r Lhey were being 
develop'^d would be of far greater service to education. This general 
call to reconceptualization of evaluation stimulated by Cronbach led 
to a reformulation of the purposes of evaluation and to an examination 
of the methodologies and the premises upon which evaluation was conducted. 

During this period of time the Phi Delta Kappa frar niity 
commissioned a group to formulate a new approach to educational p^valuation. 
This group headed by Daniel Stufflebeam and Egon Cuba formulated a decision 

"Cronbach , L. J. Course improvement through evaluation. Teachers 
College Record , 1963, 64 




131 



orientation to evaluation. The publication of their book Educational 
and Evaluation Decision Making (Stuff lebeam, 1971), has made a signifi- 
cant input to the field and strongly influenced the development of DOES. 
Another important event during this period resulted in the attempt to 
bring to educational evaluation perspectives from many other social 
science disciplines. Thus during the latter half of the sixties, a 
great effort in education was made to synthesize methodologies from 
diverse fields in an attempt to look at evaluation methodologies from 
multiple perspectives. This entire evaluation movement of the sixties 
has significantly influenced the development of the DOES system. 

Summary 

The preceding provicl^^o an overview of the forces which have led 
to the current concern for evaluation of impact of social programs and 
those forcas that have shaped this project. The evaluation problem 
for DOES is somewhat limited in that we will attempt to trace "community 
impact" as a function of training intervention. That is, we will be concerned 
only with changes in a community insofar as those changes can be related 
directly or indirectly to training conducted by the Texas Department of 
Community Affairs. 
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Community Impact Training Evaluation 
Perspective 

It is important to note that community impact evaluation as 
used in the model finds its origin in training evaluation and attempts 
to trace the effects of training through job performance to its effect 
upon the community. That is, "impact evaluation" will be concerned only 
with the extent to which training intervention and the resultant im- 
provement of job performance could be traced to some indicator within 
community life related to training and subsequent job performance. For 
example, if on the basis of a documented need receptionists the offices 
of city departments were to be trained to handle citizen te ephone complaints 
more courteously, then the subsequent job performance of the receptionists 
could be monitored and a direct or indirect measure of the effect of their 
behavior upon citizens could be ascertained. Impact evaluation could be 
conducted by means of survey methods such as post cards sent to citizens who 
called in complaints requesting an evaluation of the manner in which their 
complaint was ha .died. Another method might involve inspecting records to 
determine the trend in numbers of complaints received about courtesy of recep- 
tionists' responses to the public over time. It is important to under- 
score once again that for the purposes of DOES the community indicator must 
bear some direct relationship to training. 

Selection of Indicators 

Probably the most important problem facing the Texas Department of 
Community Affairs with respect to community impact training evaluation 
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will be the identification of community indicators that would serve as 
criteria for evaluation. The scope of the training offered by TDCA is 
so generally broad, that it may be difficult to trace objectives 
of training to a set list of community impact indicators. In all 
probability, the indicators would have to be idiosyncratic to the 
problem. In all likelihood the best and most reliable data to evaluate 
community impact as a function of training would be those public data 
that are independently gathered as municipal records. For example, if a 
municipality keeps records on requests for service, or complaints from 
the public and if all municipal departments keep records indicating 
the tasks performed by workers within departments, these records 
provide prime data for evaluation related to training and community 
impact. 

When using material process records to evaluate the effect of train- 
ing intervention, one would look at the direction of changes in indicators, 
as a function of job training, as opposed to looking for the magnitude of the 
change. There are so many confounding variables operating in a municipal 
work situation that attempts to isolate the magnitude of change as a function 
of training intervention will be futile. 

It is critically important that indicators selected for outcome eval- 
uation have credibility in the eyes of decision makers. If indicators 
are selected to satisfy primarily the needs of social scientists they may 
be perceived non-relevant by governmental decision makers; unless evaluation 
data are perceived as useful, an entire evaluation effort will fail — 
given the context of DOES, 
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Ci tizen Complaints as Social Indicators 

One effective means for tracing the effects of community 
development training would be to look at citizen compla^'' The work 

of Ezra Krendel (1970) provides a system for processir. of ti en complaints 
as an indirect measure of the quality of life within a i immur ' ^y . The 
model developed by Krendel looks at response time as a major criterion 
in achieving satisfaction on the part of the citizens of the community. 
That is, the extent to which complaints can be responded to effectively 
is a measure of the quality of life in that community. Krendel' s work was 
modeled on information gathered by the Mayor's Office of Information and 
Complaints in Philadelphia, a similar model for gathering and processing 
citizen complaints will be presented in an Appendix; the system described 
will be one developed by Mayor Orville Hubbard of Dearborn, Michigan. 
Hubbard's system has proved to be an effective means of delivering quality 
service to the citizens of Dearborn. 

Me thodo logical Problems of Community Impact Evaluation 

One of the major problems associated with gathering data 
relative to impact on a community may be one of intrusiveness . That is, 
if an evaluator were to use questionnaires, surveys or some other type 
of data gathering where "outsiders" come in*-o a community and conduct 
interviews to gather data, the results are likely to be invalid. Many 
studies have demonstrated that the tendency will be for the respondents to 
tell the interviewer what he thinks the interviewer wants to hear, not 
what the responder really believes. This distortion of information is 
particularly true when the subjects of interviews are members of minority 
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groups, ethnic groups or the poor. Another problem associated with 
interview techniques are biases imposed upon data collection and 
analysis by the investigator. Consequently, relying on these sources of 
data gathering of "community impact" creates a multitude of problems. 

Perhaps an effective means of gathering relevant information would 
be to collect data unobtrusively through observation of natural phenomena 
within a community in such a way that one does not intrude or disturb the 
environment. Methodologies employed by cultural anthropologists, socio- 
logists, and recently by educational researchers, where one becomes part of 
the environment while observing it, offer some interesting data collection 
modes (Webb, 1967). As mentioned earlier the most valuable data for 
evaluation of impact of public and social programs would be to process data 
normally collected by governmental agencies. Records of various types of 
services perfoirmed, expendiirure rates for material, police records and the 
like provide a rich source and perhaps the best source of information. If 
carefully collected, documented and selectively used, these data can provide 
perhaps the most reliable data for evaluating community impact as a function 
of training. 

Wherever possible naturalistic or process data collected by agencies 
could be scanned through a variety of techniques to detect trends over time. 
One could use a number of quasi-experimental designs such as the interrupted 
time series designs proposed by Campbell (1970) to gauge whether changes in 
trends had occurred in these data. Further, if one can identify specific 
times at which training has occurred and then develop a sampling procedure 
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to gather data at specified time intervals, this probably can provide the 
most valid and reliable way of gathering impact. 
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Process Records as Indicators 

The objective of using process record data is to develop and establish 
criterion variables by means of which the effects of implementation of pro- 
gram objective can be verified. 

There are two major methodological problems in using data from 
process records in evaluating attainment of program objectives. These pro- 
blems are the relevance and sensitivity of the data to program effects. 
Relevance refers to the logical plausibility connecting the implementation 
of program objectives to the effects produced in the data. For example, it 
would be plausible to expect that doubling the number of police patrols in a 
high crime area would eventually reduced the number impact crimes and citizen 
complaints about these crimes in that area. Both the frequency of and citi- 
zen complaints about impact crimes would be relevant measures of that program 
effect. However, one would not expect the same program strategy or the same 
data to be as relevant for other types of crimes, say, embezzlement. 

Record keeping systems in fact may be in existence. Whether they 
record data relevant to specific program objectives will be a question sub- 
juct to logical analysis of plausibilities and to empirical verification, 
A process variable may pass the test of relevance but may furnish data that 
are simply not sensitive to the effect to be measured. The lack of sensi- 
tivity of such data may be due to statistical unreliability (noisiness) 
or to inadequacies in sampling the process. To be of any use, process 
record data must be sensitive to changes brought about by implementation of 
program objective. That is, there must be a reliable association between 
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the process variable (s) and the treatment effects produced by the program. 
In addition, the sampling process governing the collection of data must 
be fine-grained enough to resolve differences within appropriate time units. 
For example, a training program may have shortened a particular service 
delivery time -rom days to hours. Collection of data to the nearest day on 
such service delivery times would not reveal the difference in reduced 
delivery times. 

The goal of using process record data is the detection of assignable 
(plausible) causes that are associated with the program implementation and 
the measurement or their effects on the community. There are inferential 
and analytic problems to be addressed in using process record data. On the 
inferential side these problems include a priori construction of a system of 
plausible hypotheses and ruling out possible spurious cause and effect asso- 
ciations. The development of a "battery" of impact indicators should follow 
a sequential sampling model in which each variable is empirically investigated 
on the basis of its sensitivity and relevance in contributing to the valid 
measurement of program effects. The sequential sampling model should also 
allow for continual modification of the system of hypotheses in order to 
achieve control over spurious and confounding variables. 

On the analytic side, sampling plans should be developed to solve 
problems in the resolution of system response times already mentioned. The 
sampling plan should also take into account the appropriate trend evaluation 
model to maximize detection of impact effects (Campbell & Stanley, 1966). 
Finally, there is the analytic problem of the appropriate statistical tests 
to use in analyzing and drawing conclusions from process record data. The 
solution to the problem resides in the use of the appropriate data model 
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for performing statistical tests. For example, when process records fur- 
nish data on response times the form of distribution (data model) for these 
data rules out statistical procedures based on the normal distribution. The 
solution would be to use some form of likelihood ratio analysis, based on 
the actual response time distribution, rather than statistics based on a 
normal distribution (Krendel, 1970; Dossett, 1965), 
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recommendations to tdca regarding community 
^Envelopment training impact evaluation 

Surveys 

Jul first recommendation to TDCA regarding community iiny..*(-L 
training evaluation resides within the input evaluation mcae« We 
strongly recommend that the agency conduct a survey of municipalities 
and county governments to identify the kinds and types of records that 
are currently being kept by these agencies ♦ The subsequent development 
of a taxonomy or catalog of naturalistic records kept by these agencies 
could then be developed so that TDCA would be in a position to know 
what data would be available to conduct impact evaluation studies. 
Following this survey and a study of the type of records kept, it 
may be possible for the agency to begin to standardize some of the data 
collection mechanisms used by different municipalities such that 
comparable records would be available. 

The second recommendation follows closely upon the first. We 
recommend that TDCA closely examine the type of system developed by 
Mayor Hubbard of Dearborn, Michigan (see Appendix I ) . This system 
if installed in Texas communities would not only directly improve 
service but the data gathered could be used to assess the effects of 
community development activity over time. Naturalistic data collected 
by local government services will provide a good foundation for the 
conduct of impact studies in the future; it would be time and money 
well spent. 

lis 
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Training Package Evaluation 

Criteria for Evaluatio n Training Packages 
From a research perspective, the state of the art in training 
materials development must be termed primitive • This statement should 
not be interpreted to mean that there are not highly refined procedures 
for the development of training materials nor is there a lack of 
evaluation of training. The point to be made, simply, is that there 
are few empirically derived generalizations that one can make about the 
effectiveness of instructional media across training areas and across 
different training populations. The reason for this lack of generalizable 
information for instructional or training materials development is 
largely due to methodological problems in training and instructional 
research. In summarizing past research, Baker^ points out that there 
is little evidence to support the notion that a given media as a media 
provides for more effective learning. Briggs^ does point out, however, 
that multi-media instruction has proved superior to single media 
instruction. During the past ten years, with the emergence of educational 
research and development related to instructional product development, 
there have emerged a number of principles to guide the development of 

^Baker, Eva L. , The technology of instructional development in 
Travers, R. M. W. , Second Handbook of Research on Teaching, Chicago- 
Rand McNally, 1973. 

2 

Briggs, L. J. et al. Instructional media . Monograph No. 2, 
Pittsburgh: American Institutes for Research, 1967. 
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instructional materials; however, most evaluative studies are specific 
to a particular training program or to a particular instructional product. 

The enormous research effort that the military has undertaken to 
evaluate the effectiveness of their training programs is indeed 
commendable. However, there still is insufficient data regarding the 
critical variables that affect instructional media as they interact with 
learner variables; no generalizable statements, emerging from research, 
regarding effectiveness of specific training materials across training 
areas can be made at this point in time. Indeed, it may not be possible 
for many years to come to make such statements. However, much can be 
gained from the practical experience and evaluation of specific courses 
provided by the military. 

In the absence of generalizable empirically based guidelines to 
support training material development then the alternative is to examine 
closely high quality training materials that have undergone extensive 
evaluation and do indeed produce the desired results in learners. 
Examination of these materials will provide a basis for developing 
guidelines and criteria that could be used to evaluate the quality of 
training materials. This is the route which the GATE project chose to 
follow. We have based our work heavily upon military training development 
and evaluation, and upon educational instructional materials development 
and evaluation. In that regard, the criteria for evaluation of training 
packages does represent a vast body of literature, and the combined 
experience of the members of the GATE project staff who have engaged in 
instructicn cil research and development. 

^Popham, W. J. in Baker and Schutz, 1971 
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Before presenting the criteria for the analysis of training 
materials, it may be helpful to view the following chart which represents 
1 state jf the art approach to the assignment of different types of 
instructional media to particular types of learning objectives. This 
chart reflects very well the experience and the evaluation results 
derived from military training, and the chart is consistent with the 
existing evaluation and limited research on educational treatment delivery. 
The chart presents a matrix of instructional media by learning objectives 
and the entries in each cell of the of the chart provide ratings of 
estimated effectiveness or suitability of media to particular objectives. 

Table 8 

Instructional Media Stimulus Relationships to Learning Objectives^ 



LEARiVTNG OBJECTIVES: 



TYPE OF 

INSTRUCTIONAL 

MEDIA 


Learning 

Factual 

Information 


Learning 
Visual 
Identifi- 
cations 


Learning 
Principles, 
Concepts, 
and Rules 


Learning 
Procedures 


Performing 
Skilled 
Perceptual - 
Motor Acts 


Developing 
Desirable 
Attitudes, 
Opinions^& 
Mot iva*. Ions 


Still Pictures 


Medium 


HIGH 


Medium 


Medium 


low 


low 


Motion Pictures 


Medium 


HIGH 


HIGH 


HIGH 


Medium 


Medium 


Television 


Medium 


Medium 


HIGH 


Medium 


low 


Medium 


Training Aids 


low 


HIGH 



Medium 


Medium 


low 


low 


Audio Recordings 


Medium 


low 


low 


Medium 


low 


Medium 


Trainer 
(Sinnulator) 


Medium 


HIGH 


HIGH 


HIGH 


HIGH 


Medium 


Programmed 
Instruction 


Medium 


Medium 


Medium 


HIGH 


low 


Medium 


Demonstration 


low 


Medium 


low 


HIGH 


Medium 


Medium 


Printed Textbooks 


Medium 


low 


Medium 


Medium 


low 


Medium 


Oral Presentation 


Medium 


low 


Medium 


Medium 


low 


Medium 
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Adapted from Allen, William H., Research in Instructional Media and 
Art Education, Final Report of Uses of Never Media in Ajrt Education, 
Washington, D. C: National Art Education Association, August 1966, in 
Air Force Manual 50-2, Instructional System Development , Washington, D. C, 
Department of the Air Force, 31 December 1970. 
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Given this description of the state of the art of training and 
instructional materials development, the following criteria should prove 
useful in analyzing quality training materials. 
Criteria for the Analysis of Training Packages 

Program Goals, Learning Experiences and Evaluation . 
1- Is there an explicitly stated general goal for the training 
package in terms of learner behavioral outcomes ? 

2. Are there specific learner behavioral objectives for each 
sub-unit of the training package? 

3. Is there a clear correspondence between the general goals of 
and the unit objectives? 

4. Do the 1('/)}'ning experiences appear to be consistent with the 
U'^.it objectives? 

5. Ar^ the ve provisions to inform learners of the objectives 
vhich they ?.re expecred to acquire? 

6. Are the uarners provided with an opportunity to practice the 
unit objectives they are expected to acquire? 

7. Are provisions nvade for feedback to learners regarding their 
progress during the training? 

8. Are provisions made for feedback to learners regarding their 
achievement at the end of training? 

9. Arj self-evaluation devices built in for learners to evaluate 
their progress en route? 

10. Are performance evaluation measures available to determine 
learner achievement? 
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11. Are performance evaluation measures congruent with unit 
objectives? 

Characteristics of Instructional Materials and Activities . 

1. Are the roles for instructors clearly specified? 

2. Are the roles for learners clearly specified? 

3. Is training provided for instructors? 

4. Are the training materials diversified with respect to media 
and instructional methods? 

5. Are the training activities diversified for learners? 
6 . Are materials likely to be perceived as corresponding 

to the developmental or maturity level of the learners? 

7. Are pretests provided for learners? 

8. Can the trcinees start at an advanced level? 

9. Can they finish at different levels? 
Characteristics of Evaluation Materials . 

1. Are evaluation items and/or devices available? 

2. Are evaluation items related to learner performance objectives? 

3. Are learners given the opportunity to practice during training 
on items similar to criterion items? 

4. Are problem solving techniques used to measure learner behavior 

B- Procedures for Evaluating Training Packages 
Given the criteria for evaluating training packages in che previous 
section, the procedures for actual evaluation of training packages are a 
straight forward activity. The following steps will assist in the 
activity, 
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Step 1. Specify: training population — who is to be trained? 

length of training in hours~how long? 
general goals of training — what do you want 

from training? 
type of training--what kind of experience do you 

want for trainees? 
cost — how much can you pay? how much can trainee 
pay? 

Step 2. Define what you mean by training: 

Do you intend your training to: Yes No 

1. present information ^ 

2. develop awareness 

3. communicate theory/knowledge 

4. promote identification and application of 
principles 

5. develop an orientation 

6. change attitudes 

7 . produce skilled behavior 

Things you do to judge the quality of training packages you are 
examining: , 

Step 3. Examine and judge the goals and objectives of the training 
package in terms of: 

Acceptable Unacceptable 

1. correspondence with your 

objectives 
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Acceptable Unacceptable 

2. course length 

3. relevance to your needs 

4. scope of objectives 

5. behavioral quality of 
ob jec tive s 

6. relevance to your pocketbook 

Step 4. Examine and judge the experiences provided for trainees: 

Yes No 

Are they related to your objectives? 

Do they require active participation of 

trainees? 

Are experiences varied? 
Step 5. Examine and judge built-in evaluation 
devices: 

Are trainees made explicitly aware of outcomes? 
Are trainees given opportunity to evaluate their 
own knowledge or skill acquisition? 
Are trainees provided feedback on their 
experience ? 

Do trainees have opportunity to rate 
effectiveness of training? 
Step 6. Examine and judge background information on 
training program: 
Do developers have adequate qualifications? 
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Yes 



No 



Is instvvctor's role clearly specified? __ 

Is pre. ,ous evaluation data presented? _ 

Step 7. Summarize your judgments and decide: 
Use program as is. 

Seek modifications in the program to make it more congruent 
with your needs. 

Judge it as inadequate for your needs. 
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Guidelines for Training Materials 
Development and Subcontracting 

Development of Training Materials — State of the Art 
What can be said about the state of the art regarding the 
development of training materials has in effect been said in the 
preceding sections on the evaluation of training packages per se. 
The two are really inseparable. As noted from the previous discussion 
on training package evaluation, the development of training materials 
is still somewhat an "art form/' However, it is highly recommended 
that officials in local, state or federal government in a position 
to subcontract the development of training materials contact two sources 
for information and materials, One source would be the Civil Service 
Commission of the federal government. Civil Service conducts a wide 
variety of training activities for government positions. The 
experience of the Civil Service with respect to training materials 
development and evaluation are well expressed in their two publications,' 
In addition to Civil Service materials, the GATE project strongly 
recommends the examination of instructional materials and training 
design systems used by the military. In this regard. Smith's book 
(1971) provides an excellent overview of the position on development 
of training systems gained through years of research and development 
with the U, S, Army, Another excellent publication is the Air Force 
manual 50-2 which deals with the problems and the processes involved in 

^See addendum to bibliography. 
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instructional systems development. These two sources provide the best 
and most comprehensive treatment of the topic. The CATIi) project highly 
recomniends them. 

The guidelines which follow provide proced^ires to be used when 
subcontracting for the development of training niaterials. A government 
official in a position to subcontract developme^^t work should not do 
this without first examining the state of the a^*t position papers with 
respect to training package evaluation and a ca^*^ful examination of 
those criteria. To gain some idea of how RFP's ^re developed, it is 
recommended that the decision-maker review requests for proposal 
procedures developed by the military. The locat^^oti of Air Training 
Command in San Antonio, Texas, provides the Tex^^ Department of 
Community Affairs with a ready resource to be t^Pped for samples of 
RFP's and samples of high quality training progJ^^ms* 

The procedures to be followed in securing ^eq^jests for proposals 
for training materials development are as follows : 
Step 1. Identify potential clients. 



A. Survey colleges and universities co identify departments, 
agencies, service bureaus, and individuals that have 
expertise in your area of interest* 

B. Survey professional organizations ^elated to the general 
field and identify a contact persf?^- 

C. Survey related governmental agencies (state and federal) 
for names of individuals who could Provide you with 
potential bidders. 




ISi 



. Survey Commerce Business Dally (U. S* Government 

Publication) for list of grant awards to private rese^^ch 
and development firms doing business in yonx: area of 
Interest. 

Collate and classify information gathered above for 
re ference. 

Prepare na letter of interest. 

Prepare a letter of interest for institutions Usted 

above to ascertain thetr interest and availability to 

provide the services you are requesting. 

In the letter of interest provide the pot3entiaI clietit 

with: 

1. Brief statement of purpose of the proposed Project' 

2. Scope and length of the proposed contract, 

3. General requirements and constraints imposed on client 
(e.g. likely amount of the contract; c<5ndition6 th^^ 
must be met) . 

Ask client, if interested in project, to provide you vi^h 
a brief prospectus of his organization including: 

1. Acttial work done related to contract area. 

2. List of personnel and qualifications who are likeXy to 
be assigned to the project if contract is awarded. 

Screen letters of interest. 

Individuals and organizations who have little or no ^Xpevi^ 
ence in your area of interest should be screened out at ^hiB 
point. 
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B. Some individuals and groups nay not have proper 
qualifications for the work and should be screened out. 

Note : The development of a response to an RFP is often a costly 
experience for both the client and the sponsoring agency; careful 
screening at this stage can save all parties considerable money. 

C. Select those individuals and /or agencies who are most 
interested and best qualified to do your work. 

Step 4. Prepare the RFP. 

A. Specify the training problem in one or two sentences. 

B. Provide information on the educaf-ion, age and work 
background of the training population. 

C. State what you would like the trainee to be able to £o 
at the end of training. 

D. Specify the work task for the bidder. 

1. Specify the product you expect from the bidder and its 
characteristics (e.g. the final product will be a set 
of training materials and accompanying leader's guide 
designed to train first line supervisors in municipal 
government to deal with the handling of minority 
group grievances. The training experiences shall 
include provisions for role playing, problem solving, 
and rmall group discussion.). 

2. Specify the delivery date for the product and the 
quantity of the final product. 

Be sure that you include a statement indicating that you reserve the 
right to reject any or all bids. 
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3. Specify the number field test conditions and data 
reporting (where applicable). 
E. Specify the time schedule, budget format, and personnel 
qua lifications. 

1. Require bidder to provide a time schedule and general 
plan for product development that include key check 
points for your monitoring purposes. 

2. Provide budget format you require of bidder. 

3. Rp.quire project staff summary v/ith qualifications of 
key personnel* 

4. Specify date and time on which bids are to be received 
and opened , 



How to Rate Contract Proposals 



RFP re: 



Date: 



Name of Contractor: 



Total Rating Points: 



Items: Contractor Qualifications 



1. 



Professional qualifications 



Project manager 



Technical support personnel 



2. 



Experience in training area 



Prior development work 



Related work 



Proposal Qualifications 



1. Clarity of training objectives 



2. Congruence of objectives with training goals 



Adapted from Otto, Clavin P. and Gl^^ser, Rollin 0., The Management 
of Training , Reading, Mass.: Addison-Wesley, 1970. 
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3. Congruence of training activities with your 
specifications 

4. Diversification of training activities proposed 

5. Acceptability of training activities to trainee 

population 

6. Provisions for feedback to trainees 

Budj^e t and Time Schedule 

1. Reasonableness of cost proposal 

2. Acceptability of check points in the development 

process 

3. Acceptability of other items 

Total Rating 

Rate each of the items as follows: 

5 for quality that exceeds expectation 

4 for highly acceptable quality 

3 for acceptable quality 

2 for marginally acceptable quality 

1 for unacceptable quality 
Note: The rating scale is provided as a means to assist in the screening 
of the overall quality of proposals. In the final analysis, one should 
screen very carefully several times, those few proposals that are of 
highest quality. Final decisions for selection of a contractor, when 
there are several competing bids, may be such variables of personal 
compatibility of project officers, distance from sponsoring agency and 
the like. 
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Guidelines for Subcontracting Evaluation Studies 

Procedures for Developing RFP*s for 
Three Modes of Evaluation 

The rationale for proceeding in three modes of evaluation 
(GATE final report) and the DOES chart in the handbook 
provide the basis for developing RFP*s for evaluation studies. 
The actual steps in subcontracting are as follows: 

Step 1. 

A, Review the types of evaluation by mode in the DOES 
evaluation system chart. (See Handbook.) 

B. Examine input evaluation for each mode of evaluation, 

1. Have these evaluation activities been conducted? 

2. If not, decide whether you can perform these 
activities yourself or whether you should subcontract 
them. 

3. If the evaluation activities have been completed, 
are you satisfied with the results? Gan you work 

ith them? 

G. decisions about the types of evaluation studies you 

wish to have performed within each mode. (E.g., Job 
Performance Process Evaluation) 
Step 2. Identify potential evaluation subcontractors. 

A. Survey colleges and universities to identify departments, 
agencies, service bureaus, and individuals that have 
expertise in evaluation, 

16 3 
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B. Survey professional organizations related to the general 
field and identify a contact person. (E.g., American 
Educational Research Association, American Management 
Association, American Institutes for Research) 

C. Survey related governmental agencies (state and federal) 
for names of individuals who could provide you with 
potential bidders. 

D. Survey Commerce Business Daily (U. S. Government Publication) 
for grant awards to private research and development firms 
who conduct evaluation studies. 

E. Collate and catalog information gathered above for ready 
reference. 

Step 3. Prepare a letter of interest to send out. 

A. Prepare a letter of interest for institutions and indi- 
viduals listed above to ascertain their interest and 
availability to provide the evaluation services you are 
requesting. 

B. In the letter of interest provide the potential client 
with: 

1# Brief statement of purpose of the evaluation study. 

2. Scope, length of the proposed contract, and expected 
dollars available. 

3. General requirements and constraints imposed on 
client (e.g. conditions that must be met). 
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C. Ask client, if interested in project, to provide you with 
a brief prospectus of his organization including: 

1. Actual samples of evaluation work done x-elated to 
contract area. 

2. List of personnel and qualifications who are likely 
to be assigned to the project if contract is awarded. 

Step 4. Screen letters of interest. 

A. Individuals and organizations who have little or no 
experience in your area of interest should be screened 
out at this point. 

B. Some individuals and groups may not have proper qualif i " 
cations for the work and should be screened out. 

Note: The development of a quality proposal response to an RFP is often 
a costly experience for both the client and sponsoring agency; careful 
screening at this stage can save all parties considerable money. 

C. Select those individuals and/or agencies who are most 
interested and best qualified to do your work. 

Step 5. Prepare the RFP. 

A. Specify the evaluation problem in one or two sentences. 

B. Provide background information on the training population 
(e.g. age, educational background ,• work experience). 

C. State the type of evaluation you want conducted by mode and 
by type (e.g. training process and training outcome 
evaluation for X course). 
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D. Provide evaluator with copy of the training materials to 
be used for Instruction, 

E. Specify the work task for the bidder. (The following 
are samples:) 

1. Indicate the general type of evaluation items you 
want used. 

a. Multiple choice items. 

b. Scalable items. 

c . Forced choice . 

d. Case problem, etc. 1 

(See Exhibit Y for samples of evaluation items.) 

2. Specify tlmatahle for your review of items prior to 
use. 

3. Specify any pilot test requirements you may have. 

F. Specify time schedule, budget format, personnel qualifi- 
cations. 

Step 6r-Send out RFP's, 

Step 7-- -Review and rate proposals. 

Step 8^ -Decide on subcontractor. 

Step 9. Negotiat^oa contract based on proposal. 

Step 10. Monitor evaluation study in process according to contract. 
Step 11. ''. iw final reports to determine compliance with origi^iial 
contract. 
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How to Rate Contract Proposals' ^ 
RFP re: Date: 
Name of Contractor: Total Rating Points: 

Items: Contractor Qualifications 

I. Professional qualifications 
Project manager 
Technical support personnel 
2* Experience in evaluation area 
^rior evaluation work 
Related research or development work 
Proposal Qualifications 

1. Clarity of evaluation procedures 

2. Congruence of evaluation study with training goals 
and objectives 

3. Congruence of evaluation activities with your 
specifications 

4. Acceptability of evaluation activities to trainee 
population 

5. Provisions for evaluation feedback to trainees, 
instructors and sponsoring agency 

Budget and Tirne Schedule 

1. Reasonableness of cost proposal 

2. Acceptability of check points in the evaluation 
process 

3. Acceptability of other items 

Total Rating 

^Adapted from Otto and Glaser (1970). 
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Rate each of the items as follows: 

5 for quality that exceeds expectation 

4 for highly acceptable quality 

3 for acceptable quality 

2 for marginally acceptable quality 

1 for unacceptable quality 
Note: The rating scale is provided as a means to assist in the screening 
of the overall quality of proposals. In the final analysis, one should 
screen very carefully several times, those few proposals that are of 
highest quality. Final decisions for selection of a contractor, when 
there are several competing bids, may be such variables of personal 
compatibility of project officers, distance from sponsoring agency and 
the like . 



If the activities and steps used to implement the DOES are 
followed, as specified in the Handbook, the monitoring of evaluation 
studies is made easy. All the monitor must do is to make sure that the 
steps are b ing followed as outlined in the Handbook and in the contract. 

The judgment of quality of an evaluation study is also facilitated 
by the activity lists provided in the haildbook, for evaluatio.. by each 
mode and stage. In many instances, reference sources are suggested 
for review. These sources provide good examples oi objectives, evaluation 
instruments, and the like. A comparison of the work of the subcontractor 
with examples in these sources (e.g. the Bloom and Krathwohl Taxonomies 



B . Criteria for Monitoring and Judging the Quality 



of Att Evaluation Subcontract 
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of Educational Objectives) provide a basis for judgment of quality. 

From a quite different perspective, the sponsoring agency should 
judge the quality of an evaluation subcontract on the basis of very 
practical criteria. The practical criteria of relevance, importance, 
scope, credibility, and timeliness (Stuff lebeam, 1971) are very 
important quality criteria. 

Relevance — Are the data presented by the contractor relevant to 
the purposes intended? Do you perceive the data to be relevant? 
Importance --Are the data presented by the contractor really 
important to you? Have you been given too much information such 
that it is difficult to weed out the unimportant from the 
important? 

Scope — Are the data presented to you by the contractor (assuming th 
they are relevant and important) equal to the scope of the problem 
Leing investigated? Are the data too narrow in scope to be useful? 
Did the contractor ignore important sources of data, from your 
perspective? Are the data too broad in scope to be useful? 
Credibility — Are the data presented to you by the contractor 
believable? Will they be "believed" by your superior? Are the 
data relevant, important, adequate in scope? Are you willing to 
"stand up and be counted" with the data you have :rom the sub- 
contractor? 

Timeliness — Did the subcontractor deliver his data on time ? Was 
the timing of data delivery useful to you in the internal decision- 
making processes of your agency? 
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From the perspt:ctive of measurement and evaluf*tion specialisiis, 
the quality of an evaluation study tif^uld depend upon the criteria of 
validity and reliability. In thi<> sense, validity refers to the 
'•quality" of data, data gathering rssearch des^,gn, and ,he conclusions 
drawn from the data. Reliability refers to the consistency of the 
data gathered. Reliability is best reflected in the statement, ''If 
new data were gathered, would the results likely be the saine?" 

If a sponsor would like to place judgments of quality on an evalua- 
tion study from an evaluation methodology perspective, it would be wise 
to hire another evaluator to **audit" the work of your subcontractor. 
This would probably provide the best mean? to judge the quality of an 
evaluation subcontract. 
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Reporting of Decision Information 
The guiding principle in this study has been that evaluation is the 
process of identifying, obtaining and using information to judge, and 
select among, decision alternatives. Evaluation is a complex technical 
process that yields technical information. Yet technical information alone 
is not all that the decision maker needs. The vital link between evaluation 
and action selection is the decision information system. To* be coiiq)atible 
with the decision maker's requirements, technical evaluation information must 
be partitioned, translated and displayed in accordance with his requirements and 
constraints. Technical information needs to be translated into the meaning 
system of the decision maker. The results of this translation produce decision 
information that is: 

1) On target - decision information that provides the 
decision maker with the kind of information he needs. 

2) Of sufficient quality - information that is adequate for 
making decisions without overloading the decision maker. 

3) Time ly - decision information that is produced and disseminated 
promptly with sufficient lead time in advance of important 
decision deadlines. 

4) Sensitive ~ decision information that is responsive to 
community needs and appropriate to the requirements of 
training developers and training administrators to keep 
programs on target, and congruent with training goals. 
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The decision maker will require technical evaluation iafonn^tion in the 
form of technical reports and accompanying technical abstracts. The technical 
level of reporting presents the procedures used in evaluation, data analyses 
and summaries, instrumentation, and interpretation oi t*esults in technical 
terms. Decision information is of quite a different ^^nd and the result of 
translation of technical information into ratings of ^tainirig quality and 
includes cost data as well. 

To accomplish effective translation and reporti^S of decision infornia- 
tion will require involving the decision maker in the design of the reporting 
system and the decision information formats. 

Pilot formats for reporting decision information^ have been developed. 
These are the Decision Information Summaries (DIS) sh*^^ in Figures 3 through 

7 . The partitioning of information is in terms o£ ^he evaluation modes. 
There are two forms for training mode evaluation: Tr^itxiJ^g M;aterials Evaluation 
(Fig. 3) and Training Process Evaluation (Fig. 4). t^^re is one DIS for the 
Job Performance mode and one for community impact (Pi^s, 5 ^^d 6)* Finally, 
there is a DIS for compiling the results of the multi^^ode evaluation into a 
single format to provide a comprehensive summary of qi^^lity, costs 5 and reco- 
mmendations for specific training activities. 

The iuformation for detemining quality ratings -for Uie decision inSoxraa- 
tion summaries comes from the application cf criteria guidelines for 
evaluation. Pilot criteria and guidelines have been developed for training 
materials and training process evaluations, but have ti^t yet been explicitly 
developed for job performance anri connnunity impact. ^I^ch Dls for each mod^ 
has several parameters of evaluation highlighted in tl^e boxes on the left of 
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each DIS format. For example, the DIS for Training Material^ Evaluation 
the following parameters: needs assessment , target population , trairjing^^go^]^, 
behavior al objectives , design of package , and evaluation Guidelines , Quality 
ratings are applied to each parameter for each mode of evaluation- The r^^lngs 
are on a seven point scale from one (inadequate) through four (tnargin^l) to 
seven (adequate). Ratings are to be based on the judgments of the evalUatoi?s 
from applying the criteria and guideline. Each DIS includes ^ graphic tatitig 
scale (from one to seven) for overall evaluation for that mode, the oVer^H 
evaluation of quality is based on a weighted average of the ratings on each 
parameter within the particular evaluation mode. The DIS for comprehensive 
evaluation has as its parameters the ratings from each of the four decision 
information sumnaries. The final comprehensive evaluation is hn<^ed on the 
w*: >ht:ed average of the overa 1 quality ratings from the four <^^cision infoir^ 
mation summaries for the three modes of evaluation. The total costs for 
training and evaluation for the particular trainin:^ activity will be included 
on the comprehensive evaluation DIS. 

Finally, attached to the DIS for comprehensive evaluation vill be ^ 
summary of the major technical results indexed to the technical IrepOft ar^d a 
list of specific recommendations indexed tc the decision information suniojaJ^i^s 
for aach evaluation mode. 
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Summary and Recommendations for Future Development 

The GATE project has focused on the development of an evaluation system 
for Title VIII community development training. Phase I, system development, 
was the result of 3.6 man-years of effort culminating in a prototype system 
design which has been partially pilot tested. During the eleven months of 
the project, members of the project staff visited 38 sites conducting train- 
ing and evaluation; contacted and interviewed 55 professional individuals 
".nvolved in training system development and evaluation, and reviewed 878 
research and development documents related to training and evaluation. 

The development of the evaluation system is described in this final 
report. The tangible procedures for application of the system have been 
brought together in the form of a users' handbook (Deoision-Oriented Evalua- 
tion System: Handbook fov Community Development Training Eval .uztion) (or 
DOES Handbook for short). The DOES handbook addresses evaluation problems 
in the total training system context. This system context perspective sees 
training and evaluation as highly integrated, and indeed inseparable, parts of 
a larger system which begins with community needs assessment and ends with 
program action decisions. The evaluation principle is not to determine 
whether any training vor the trainee) is good or bad, but rather what spe- 
cific actions can be taken to deliver what is needed to the sys'cem. From 
the perspective of thi project, evaluation is seen as a collection of dynamic 
and adaptive process mechanisms thiit provide continuous f^-edback to all 
components for modification and improvement of total system performance. 
Methodologies have been researched and translated in the handbook into 
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applied procedures for evaluation in the following moc- ; 



Mode 1. 



Training Evaluation which has two sub-modes 



a. Training Development Evaluation 



Trainiiig Process Evaluation 



MoH- 2. 



Job Performance Evaluation 



Mode 3. 



Cormunity Impact Evaluation. 



Overlaying the thrae modes c 



o^ition are three phasic components: input 



stage, process stage and outc^.>r . age. The input stage refers to those pro- 
cedures and .>teps whicVt should be taken by trainees and evaluators prior to 
the actual process training. These procedures and step£ are to insure stan- 
dards for bota •^ralni:')^ and evaluation before the program is executed. The 
process si: age einbodies procedures for evaluation while activities are in action 
during training, on the job, and in the community. Finally, the net result of 
training active tiey are evaluated through procedures set forth in the outcome 
stage. Feedback infornnation is returned to the other system conq)onents, es- 
pecially deveJ.opers and training activity administrators, during each stage. 
Ttiis modeo-by- stages conceptual organization generates a nine-cell matrix of 
procedures which is the cere of operationalizing the evaluation syste n (DOES). 

Since evaluation takes place within the context of a larger sys ^en), 
operationalizing -he evaluation components cannot be fully effective unless 
problems associated with the other system components are addressed. There- 
fore, consideratic nust be given to developing effective needs assessment, 
performing job and task analyses, contracting guidelines for training develop- 
ment and evaluation development, and translation of technical information 
to decision information. Fro: the DOES perspective these latter components 
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are supporting components and are precursors to the ultimate effective 
devcslopment of DOES. 

In summary, the operational procedures for system application have 
been developed for the evaluation conqjonents as well as the supporting com- 
ponents of the total system. These operational procedures along with a variety 
of evaluation and decision-making guidelines are organized and presented in 
prototypic form in the DOES handbook. Since the system is at the prototype 
stage, further devblopment and testing of course will be required before 
final operationalization and installation. However, the DOES handbook is 
sufficiently de'^^ploped at this stage to carry out such operational field- 
testing. It is anticipated that such field testing will produce significant 
modifications in the handbook. Modification is to be expected and welcomed 
simply because the system has not yet beeu tried out in an operational 
setting. 

Even though the system has not been tested there are a number of sub- 
stantive recommendations based on the project's research effort that can 
be put forth at this time. 

1. Training evaluation . rie most productive area of inquiry has 
been the field and training of instructional systems development and evalua- 
tion. Tliere is a vast body of literature on training and instruction and 
a great deal can be translated and applied to TDCA training problems. It is 
highly recommended that efforts be concentrated in this area tc achieve high 
quality training. Improvement in training quality would lead to more effective 
job performance measurement and provide better opportunities to trace train- 
ing effects into impact on the community. 
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2. Job performance evaluation . Job performance measurement is a complex 
endeavor to say the least. Effective measurement of the transfer effects of 
training; to the job depends on a number of interrelated variables. Among these 
are the level of specific skill ::raining, the length of the training course, 
the amount of skill practice in trainin^, the intervening time between train- 
ing and measurement, and, most important, the method of measurement. In 
general, training activities that are not focused on specific skill training 
are ...ost difficult to evaluate in the job performance mode because of dilut'on 
and consequent difficulty of observing transfer of training effects. The 
same is true for training that is of short duration or does not provide skil 
practice. The problem with intervening time between training nnd evaluatien 
is that the longer this time the more opportunity there will be for causes 
other that, the specific training activity to exert themselves, such as 
informal on-the-job instruction. 

The most serious problem in measuring job performance resides in the 
kind of measuring instrument to be used. The ideal and most effective means 
of Ttieasuring job performance are performance tests. The least effective are 
descriptive pinions. Somewhere in between lie rat±uix scales. Well con- 
structed performance tests allow for objective observation and measurement 
of specific task performance. They can be, and often are, sturdy and 
effective br-^dges between n ling and on-the-job performance by insuring 
consistency between the performance requirements of the job and training 
objectives for that job. Performance tests provide the most valid criteria 
for evaluating and modifving "raining. Performance tests have another large 
pay-off: they ar excellent training devices. On the other hand, per for- 
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mance tests are costly ia tvirjna of txi:/d and materials to develop. They 
are also costly to admini er. Except for their use in ._ae araed services, 
the costs of performance tests have pre 'nted wide-scale usage 

Unfortunately most evaluation of job performance falls back on the 
use of rating devices. Ratings, while the least expensive means of per- 
formance measurement, are also among the least valid. 
Two recommendations are put forth: , 
(a) Where possible, investment should be made in developing 

and constructing inexpensive performance tests; and 
(b' If rating scales are to be used, the evaluator should 
base these scales on the analyses of job tasks and 
insure that the raters are well trained. 
3. CommuTiity impact evaluation . While there has been a great deal 
of theci7 written in the past decade on the topic of community impact indica- 
tors within the social science field, little of Liis is on the practical 
side. ITie problen^ of achieving effective impact evaluation is still fra^ 
with .T^ethodologlcal problems from an evaluator's perspective and with defini- 
tional prc:;jcviiv=i frov the social scientist's perspective. In practical terma, 
'ht.'z 1. rV^ :*n.'i.ei"i.oiv problem of what to measure that is specifically asso- 
ciated wit'; training. T'ue problem is similar to the problem of measuring job 
perf ormtinc . c ilv mor-: 3oriou.*b - 

The rc commendation fo) ouusrnunity impact measurement would be to use 
wheie possible natural process records. Such records are often kept by 
government agencies and offer the best existing framework, currently, for 
data gathering on impact evaluation. While most of the daua available in 
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y ^aient agencies may appear to be indirect to the problems such data are 
illy easily accessible, and until the field becomes more sophisticated, 
chey offer good prospects. 

Analyses of citizen complaints can provide a good measure of quality 
of life in a community. To be recommended is the development of such sys- 
tems as those employed by Mayor Hubbard of Deai.L>om, Michigan and the 
Mayor's Office of Information and Complaints in the city of Philadelphia. 

In summary, the research conducted on the GATE project to date allows 
the listing cf the following recommendations for TDCA: 

1. Training ; employ criteria and guidelines for evaluation 
ot training developiaent and t' '.ning process to improve 
the quality of training. 

2. Job performance ; develop performance tests as means of 
obtaining valid measures of transfer of training effects, 
to insure consistency of job performance requirements 
and training objectives, and to provide learning tools. 
Use task-specific rating scales. 

3. Got nity impa ct: investigate natural process records as 
a criterion source for evaluating training as it affects 
the community. Develop a syster. for obtaining and ai^a yf^-- 
ing citizen complaints for a -sure of training effec- 
tiveness as it affects the quality of life in the community. 



184 



BIBLIOGRAPHY 



Abt, C, C. An approach to methods of combined socio technological 

forecasting. Technological Forecasting and Social Change , 1970, 
_2, Yl^ll. 

Air Training Command. Guide for special training. ATC Manual 52-9, 
Randolph AFB, Texas: Department of the Air Force, Headquarters 
Air Training Command, January 31, 1972. 

Axr Training Command. Planning, preparation, and quality control of 

career development courses. ATC Manual 52-2, Randolph AFl, Texas: 
Department of the Air Force, Headquarters Air Training Command, 
December 15, 1972 (Effective January 1, 1973). 

Air University, Directorate of Curriculums. The use of technology to 
instruct institutions of higher education. M-42252-4-U , AUL/LRX, 
Maxwell AFB, AL 36112, 1970. 

Air Univ-:. i.ty. Directorate of Evaluation and Special Studies. S^imraary 
of Che. present AFROTC curriculum. M-42253-3-U, AUL/LRX, Maxwell 
AFB, AL 36112, 1967. 

ki ' Univei's ' L-y, u} i .torate of Evaluation and Special Studie/i, Survey 

of £r Lo-.'::c> nir lap'^ment education and training. H-42253"~U, AUL/LRX, 
Ma:'wel< yP. AL '^^il2, 1967. 

Air Univer^ ^ Extension. Course Institute. Management for air force 
supervisors. Vol. 2, NGT-jI^I?, 00006 02 1061 0667, Gunter AFB, 
Alabama . (undated) 

Aii. university Study Group. Twenty years oi: reading instruction in the 
Air University. 372, 413, A2982t, AUL/LRX, Maxwell AFB, AL 36112, 
1968. 

Alberts, D. S. A plan for measuring the performance of social programs. 
New York: Praeger Publishers, 197 0. 

Alberts, D. S. An operations research approach to measuring the performance 
of social programs. A Paper Presented at the Annual Meeting of th^ 
Ohio Valley Sociological Society, Cleveland, Ohio, April 1971. 



185 



2 



Albrech:, G. L. Agents of social change and social conflict in 
developing countries: Indigenous health professionals and 
paraprofessionals in Iran and Kenya. Atlanta: Emory University, 
1972. 

Allen, B. D. et al. An analysis of regional centralization of base 
i /el logistics computer operation. SLSR-54-67. AFIT/L, 
Wright-Patterson AFB, Ohio, 45433, 1968. 

Allen, J. R. An analysis of aircrew coordi .ation. M-35582-7-Ur 

A 42;ia, Research study. AUL/LRX, Maxwell AFB, AL 36112, 1972. 

Allen, L. A. Principles of program planning for tra'ning adults. 

Presentati( . at the Educational Program for Public Library Per- 
s.r-mel. University Kentucky, February 1969. (ERIC ED 045 950.) 

Allerhand, M. F. et al. h study of impact and effectiveness of the 
comprehensive manpower proj ct of Cleveland. (AIM-JOBS). Vol. 
1. Summary. Vol. 2. Technical Report. Vol. 3. Appendices. 
Cleveland: Case Western Reserve University, December 1969. 
(ERIC ED 042 117.) 

Aman, E. D. Flying safely in spite of the birds. M-35582-7-U, A 
484f , Research Study, AUL/LRX, Maxwell AFB, AL 36112, 1972. 

;^jranentorp, W. et al. Training for development and evali^ation in educa- 
tion. The proposal for the midwest educational training center. 
Contract no.. OEC-3-7-06 28 70-3069, Minneapolis: Upper Midwest 
Regional Educational Lab, Inc.j December 1970. (ERIC ED 056 413.) 

Anderson, C. A. Effectiveness measures. M-32983~U, /'^451e5 Professional 
study no. 3266. AUL/LRX, Maxwell AFB, AL 36112, lto7. 

Anderson, C. Mc et al. Approaches to the analysis of social service 

systemLi. In E. E« Schwartz (Ed.), Planning-programming - bud ge ting 
- systems /^.ixii socl£l v;elfare . Chicago: The University of Chicago, 
TYr- Sch:oI ot Social Service Administration, 1970. 

Anderson, C. T. The tactical fighter pilot: Improving combat effective- 
ness. M-32^83-U, A5-^32r.j Professional s'ludy, AUL/LRX, Maxwell AFi>, 
AL 3 12, 1972. 



18G 

EKLC 



3 



Anderson, J. G. Social indicators and second order consequences: 
Measuring the Impact of innovative health and medical care 
delivery systems. Research Memorandum no. 71-7, Lafayette, 
Indiana: Purdue University, School of Industrial Engineering, 
1971. 

Anderson, J. G. Causal models and social indicators: Toward a social 
systems model. Lafayette, Indiana: Purdue University, 1972. 

Anderson, W. How ef f ect ^'e o human relations training for the first- 
line supervisor. Personnel , 1963^ 40 , 62-66. 

Andrews, R. L. Utilization of a seli^ction criteria for screening 
applicants for administrative l:raining for the predicti n of 
^>atLe:rua of predecisional behavior at the end of the training 
perioJ, Paper Presented at the American Educational Research 
Association Annual Meeting, Minneapolis, Minnesota, J970. 
(ERIC ED 041 390.) 

Andrzejewski, N. Social indicators and accounting: Bibliography and 
V rh in progress. Unpublished manuscript, Ithaca, New York: 
Cornll University, Department of Rural Sociology, 1970. 

Argyris, C. Explorations in interpersonal competence-I . Journal 
of Applied Beh vioral Science , 1965, 1^, 58-83. 

Arkansas Univc. .. cy. College of Education. Guide to improving vo- 
cational education evaluation. Fayettville, Arkansas, December 
970. (ERIC ED 047 153.) 

Armstrong, G. R. & Dougherty, P. J. A study of the development of 
output measures. SLSR-24-71A, AFIT/L, Wright-Patterson / 
OK 45433. 

Armstrong, R. H. & Kobson, M. The use of games in planning. Long 
Raage Planning , 1972, 5 (1), 62-66. 

Ar.iola, j".. K. A revised training and evaluation program for SAC 

minuteman fissile combat crew members. M-35582-7-U, A756r, Re- 
search study no. 0075-71, AUL/LRX, Maxwell AFB, AL 36112. 



187 



As^-'^^'^r, C. The administration of publicly-aided housing. Public 

Administration , London: Journal of the Royal Institute of Public 
Administration, Autumn 1972. 

Aslin, C. H. An analysis of aircrew standarc 'zation in the Strategic Air 
Command. M-35582-7-U, A835a, Thesis no. 0055-67, AUL/LRX, Maxwell 
AFB, AL 36112, 1967. 

Baker, K. D. So you want to be a supervisor? Bureau of Narcotics and 
Dangerous Drugs Personnel Management Series , 1972, _16^ (1). 

Baker, M. R. United States Air Force physical fitness: Past, present, 
and future. M-35582-7-U, B168a, Thesis no. 0065-68, AUL/LRX, Max- 
well AFB, AL 36112, 1968. 

Baker, R. L. Educational innovation: Research and evaluation techniques 
Report of 1967 Summer Institutes, Ingl^^.-^ .od, California: Southwest 
Regional Educational Lab., 1970. (ERI. ED 044 387.) 

Bass, B. M. & Vaughau, J. A. Training in industry: The management of 

learning. Monterrey, California: Brooks Cole, Division Wadsworth 
Pub Co., 1966. 

Bass, D. D. The development of future air force managers. ri-32983-U, 

B3171d, Professional study no. 4070, AUL/LRX, Maxwell A^B, AI 36112 

Batchelor, P. G, Faith and farm: Community orientated rural training 
in Nigeria. Community Development Jo rnal , 1970, _5 (2), 79-84. 

Bauer, R. A. (Ed.) Social indica tors. Cambridge, Massachusetts: The 
Massachusetts Institute of Technology Press, 1966. 

Bauer, R. A. Social indicators and sample surveys. Public Opinion 
Quarterly , 1966, 30, 339-352. 

Bauer, K. A, Societal feedback. The Annals of the American Ac demy of 
Politica- ar S ocial Sci t xe, 1967, 373, 180-192. 

Bauer J R. A. Social indicators: Or, working in a society which has 

better social statistics. In S Anderson (Ed,)^ Planning for diver- 
sity and choice : Possible futures and their relations to the man- 
controlled environment . Cambrid;^e, Massachusetts: Massachusetts 
Institute of Technr^ogy Press, 1968. 



B.'Hiijr, K. A. h, liauer, A. 11. AmerJca, maHs Hocloty and inann mrdla. 
The Journal (^f Social iflsueB , lOdO, J^, 3-()(). 

Bauman, D. J. et al. A guide to the instruments developed for 

evaluation of in-service institutes. April 1970. (ERIC ED 
055 792.) 

Bauman, Z. Uses c"* information: When social information becomes 

desired . The American Academy of Political and Social Science , 
1971, 393 , 20-41. 

Bayton, J. A. & Chapman, R. L. Transformation of scientists a.id^ 
engineers int^ managers . NASA SP-291, Washington, D. C: 
NASA. 

Beal, G. M., Brooks, R. M. , Wilcox, L. D., & Klonglan, G. E. Social 
indicators: Bibliography I, sociology report no. 92. Ames, 
Iowa: .wa State University, Department of Sociology and Anthro- 
pology, 1971. 

Beal, G. M. , Klonglan, G. L., et al. Social indicators and public 

policy: Toward an alternative approach. A Paper Presented at 
the Annual Meeting of the American Sociological Association, 
Denver, Colorado, September 1971. 

Bell, D. The measurement of knowledge and technology. In E. B. 

Sheldon & W. Moore (Eds.), Indicators of social change : Con- 
cepts and measurements . New York: Russell Sage Foundation, 
1968. 

Bennett, M. K« On measurement of relative national standards of living. 
Quarterly Journal of Economics , 1937, February, 317-335. 

Berdie, R. F. Consistency and generalize jillty of intra-individual 
variability. Journal of Applied P -ychulogy , 1969, 53, 35-41. 

'^ernstein, B. R. & Gonzales, B. K. L^.arning, retention, and transfer: 
I. NAVTRADEVCEN Technical Report, 68-C-0215-1, 1971. 

Berry, N. H., Nelson, P. D., & McNally, M. S. A note on supervisor 
-atings. Personnel Psychology , 1966, 19, 423-426. 



18£ 



Bhola, H. S. Innovation researcl; and theory. U. S. Office of 
Education contract no. OE-5-10-307, Columbus, Ohio: Ohio 
State University, School of Education, Educational Development 
Division, 1965. 

Bhola, H. S. Configurations of change, an engi'ieering theory of 

innovation diffusion, planned change and davelopmeut . Blooming 
ton, Ir:diana? Indiana University^ oOtapcTntive Education 
Department, i. 7^ ; 

Biderman, A. D. National goals and statistical indicators. Wash- 
ington, D. C: Bureau of Social Science Research, inc., 1963. 

Bi.ierman, A. D. Kino^'atics for social indicators. Washington, D. ( 
Bureau of Social Science Research, Inc., 1971. 

Bird, L. L., Jr. Air force reserve officers* training corps: An 

analysis of its problems with cost and responsiveness. M-35582- 
7-U, B618a, Thesis no. 0090-68, AUL/LRX, Maxwell AFB, AL 36112, 
1968. 

Bixhorn, H. Cluster analysis: An application to typology of urban 
neighborhoods. A Paper Presented at the Annual Meeting of the 
American Statistical Association, Fort Collins, Colorado, 1971. 

Black, D. & Bottenberg, R. A. Comparison of technical school and on- 
the-job training as methods of skill upgrading. D3ui. 45/27, 
70-48, San Antonio, Texas: Lackland AFB, Personnel Division, 
December 1970. 

acl'vWeli, N. An analysis of the impact of advancing electronics 
techno] r7.v on air force technician training. M-35582-7-U, 
B632^ arch study no. 0120-70, AUL/LRX, Maxwell AFB, 1970. 

Bla-Lrdell, T. C. Problems of evaluating the effectiveness of develop 
ii 3nt measures. Economic Developmeni, and Cultural Change , 1954, 
2, 286«.297. 

Blumbe.j, H. H. , DeSota, C B. , & Kuethe, J. L. Evaluation of rating 
scale formats. Personnel Psycholc :;y , 1966, 1£, 243-239. 



190 



7 



Blumenthal, M. D. Alienation and violence for social change. A 

Paper Presented at the 13th Canadian-American Seminar Program, 
Windsor, Ontario, Canada, November, 1971. 

Blumenthal, M. D. & Andrews, F. M Resentment and suspicion. Ann Arbor, 
Michigan: The University of Michigan, Institute for Social Re- 
search, 1970. (a) 

Blument'^al, M. D. & Andrews, F. M. Valves: Retributive justice, self- 
defense, kindness. Ann Arbor, Michigan: The University f 
Michigan, Institute for Social Research, 1970 (b) 

BlLijisnthal, M. D., Kahn, R. L., & Andre Fo M. Attitudes toward 
violence. Ann Arbor, Michigan: The University of Michigan, 
Institute for Social Research, 1971. 



Bohrer, R. R. The air force and human relations. M-35582-7-U, B677a, 
Thesis no. 0205-67, AUL/LRX, Maxwell AFB, AL 36112, 1967. 

liolby, R. L. & Schriver, W. R. Norn/cge benefits of v:icational training: 
employability and mobility. Industrial and Labor Relations Review, 
1970, _23 (4), 500-509. 

Bolton, C. K. & Corey, K. E. A selected bibliography for the training 
of citizen-agents of planned community change. Revised Edition, 
Exchange B'bliography /^206, Monticello, Illinois: Council of 
Planning Librarians, August 1970. (ERIC ED "56 957.) 

BonJea , C. H., Hill, R. J., & Melemore, S. D. Sociological measurement: 
inventory of scales a id indices. San Francisco, California: 
Chandler Publishing Co., 1967. 

Boos, A. G. Increased pilot production: Where do we go from here? 

M-35582-7-U, B7 242i, Research study no. 0135-70, AUL/LRX, Maxwell 
AFB, AL 36112, 1970. 

Boruch, R. F. & Creager, J. A. A note oa stab^'ity of self-reported 
protest activity and attitudes. Measurement and Evaluation in 
Guidance , 1972, ^ (2), 332-338, 

I'oruch, Ro F. & Wolins, L. Procedure for estimation of trait, method, 
and error variance attributable to a measure. Educational and 
Psychological Measurement , 1970, 2£> 547-: \. 

191 

o 

ERIC 



8 



Boruch, R. F., Larkin, J. D., Wolms, L., & MacKinney, A. Aluernative 
methods of ui:. lysis; Multi trait-multi method data. Educational 
and Psycno : ogic al Measurement , 1970, 30, 833-853. 

Borus, M. E. & Bu;>:::, C. G. Problems and issues in the evaluation of 
manpower programs. Industrial and Labor Relations Review , 1972, 
25 (2), 23h 45. 

Boucher, W. I,. The future environment for technology assessment. In 
M. J. Cetron & B. Bartocha (Eds.), Technological assessment in 
perspective . Paris, France: Hemv^ra'i Prefis, 1971; 

Boyd, B. B. & Jensen, J. M. Percepti^m^ o" he first-line supervisor's 
authority: A study i superlo -subordinate communication. 
Academy of Management Journ ai "^2» 15 (September), 331-342. 

Brady, E. L. & Branscomb, L. M. ' :ion for a changing society. 

Science, 1972, 175, 961^-9i 

Brady, H. Social indicators: Ke-port bibliography, Washington, 

D. C: National Science Fr >^^acion, Division of Social Sciences, 
1970. 

Bratt, E. C. Economic growth and fluctuations. The Annals of the 

American Academy of Political and Social Science , 1971, January, 
121-131. 

Brecher, J. D. Improvement of tactical fighter continuation training. 
M- ' 582-7-U, B8291i, Research study, AUL/LRX, Maxwell AFB, AL 
36112, 1972. 

Breckner, W. J. & Baird, 0. B. Programmed instruction for operational 
aircrew continuation ground training: Adjuuv'it to self-study. 
M-35582-7-U, B829p, Research study no. 0185-70, AUL/LRX, Maxwell 
AFB, AL 36112, 1971. 

Bristol, C. D. , Jr. An improved flight eva. ation system for SAC. 

M-32983-U, B8"Di, Prof essic;ial stnu,^ no. 3884, AUL/LRX, Maxwell 
AFB, AL 36112, 197C. 

British Ministry of Labor, Central Tr ining Council. Development of 

managers — furcher proposals. Report by the Centrf*L Training Council 
of the Management Training and Development Committee, London, 1969. 
(ERIC ED 031 647.) 



192 



9 



Brittain, C. New technology and the air force transportation officer — 
will he be equal to the challenge? M-35582-7-U, B862n, Thesis no. 
0260-67, AUL/LRX, Maxwell AFB, AL 36112, 1967. 

Bronnenberg, G. E. & Golly, R. T. An analysis of mils trip training and 
its effect on milstrip requisition errors. SLSR-6-67, AFIT/L, 
Wright-Patterson AFB, OH 45433, 1968. 

Brooks, R. M. , Wilcox, L. D., Beal, G. M,, & Klonglan, G. E. Toward the 
measurement of social indicators: Conceptual and methodological 
implications. Procee^'ings of the Social Statistics Section, 
Washington, D. C.: American Statistical Association, 1971. 

Brubaker, R. D. They're in the army now. M-32983-U, B886t, Profession- 
al study, AUL/LRX, Maxwell AFB, AL 36112, 1972. 

Bruno, J. E. A methodology for the evaluation of instruction or 

performance contracts which incorporates school district utilities 
and goals. American Education Research Journal , 1972, 9^ (2), 175- 
195. 

Bryan, C. H. & Bertrand, A. L. Propensity for change among the rural 
poor in the Mississippi Delta: A study of the roots of social 
mobility. Agricultural Economic Report no. 185, Washington, D. C: 
Economic Research Service, U. S. Department of Agriculture, 1972. 

Budd. R. Technology unemployment. M-32983-U, B927t, Professional study, 
AUL/LRX, Maxwell AFB, AL 36112, 1972. 

Bureau of Labor Statistics. Planning and administration personnel in 
local governments. A Pilot study. Report no. Bulletin-1631 
Washington, D. C, June 1969. (ERIC ED 040 282.) 

Bureau of National Affairs, Inc. Training employees. Washington, D. C, 
1969. (ERIC ED 038 608.) 

Burke, D. J. An investigation of the use of the case method in performance 
appraisal training. M-35582-7-U, B9591i, Thesis no. 0280-67, AUL/ 
LRX, Maxwf-11 AFB, AL 36112, 1967. 

Burns, J. J. A summary report of the state of New York for the years 

1969-71. Albany, New York: Office for Local Government, Executive 
Department, May 1971. 



193 



10 



Burton, K. E. Improved ground training for SAC aircrews. M-35582- 
7-U, Br'74'Ji, Research study no. 0210-70, AUL/LRX, Maxwell AFB, 
AL 36112, 1971. 

Bush, J. W. & Fanshel, S. Basic concepts for quantifying health status 
and program outcomes. Unpublished manuscript^ San Diego, Cal- 
ifornia: University of California, 1970. 

Butler, W. P. An analysis of the policies and p'-ocedures for the basic 
operational training of coast guard aviators. M-32983-U, B9871a, 
Professional study no. 3309, AUL/LRX, Maxwell AFB, AL 36112, 1967. 

Byers, K. T« (Ed.) Employee training and development in the public 
service. Chicago: Public Personnel Association, 1970. (ERIC 
ED 049 421.) 

Byram, H. M. A five-state try-out and demonstration program to deter- 
mine the generalizability of an evaluation sys'i:em for local pro- 
grams of vocational and technical education. Final report. Contract 
no. OEG-0-8-070968-2829, East Lansing: Michigan State University, 
College of Education, February, 1971. (ERIC ED 049 386.) 

Cabello, 0. The use of statistics in the formulation and evaluation of 
social programmes. Proceedings of the Socrl .:.! S cat is tics Section, 
American Statistical Association , Washington, C, 1959. 

Cain, G. G. & Hollister, R. G. The methodolo^i;y of eventuating social 
action programs. Madison: Wisconsin University, C^^nter for 
Studies in Vocational and Technical Education, November 1969. 
(ERIC ED 044 611.) 

Cain, K. M. Aircrew training costs vs. full mission simulator. M-32983- 
U, C1351a, Professional study, AUL/LRX, Maxwell AFB, AL 36112, 1972. 

Calouste Gulbenkian Foundation. Community work and social change : A 
report on training . London: Longman Group Ltd., 1970. (ERIC 
ED 045 889.) 

Calta, D. An evaluation of the initial level of preparation of today's 
logisticians. SLSR-18-71B, AFIT/L, Wright-Patterson AFB, OH 
45433, 1972. 



194 



11 



Campbell, D. T. Reforms as experiments. American Psychologist , 1969, 
24, 409-429. 

Campbell, D. T. Considering the case against experimental evaluations 
of social innovations. Administrative Science Q uarterly, 1970, 
15 (1), 110-113. 

Campbell, J. P. Personnel training and development. Minneapolis: 

Minnesota University, Department of Psychology, June 1970. (ERIC 
ED 042 348.) 

Campbell, J. P. Personnel training and development. Annual Revie^J of 
Psychology , 1971, 22, 565-595. 

Campbell, J. W. An analysis of system program training. M-35582-7-S, 
ClBBa, Thesis no. 0325-67, AUL/LRX, Maxwell AFB, AL 36112, 1967. 

Candeub, I. Proper training for planners. Planning 1966 , Chicago: 
American Society cf Planning Officials, 1966. 

Carlton, S. T. & Braulc, M. B. In-basket scoring manuaJ.. Princeton: 
Educational Testing Service, December 1971. (ERIC ED 059 275.) 

Caro, F. G, (Ed.) Readings in evaluation research . New York: Russell 
Sage Foundation, 1971. 

Carrol, S. J. & Taylor, W. H. A study of the validity of a self- 
observational central signaling method of work sampling. Personnel 
Psychology , 1968, 21, 359-364. 

Carron, H. J. The need for a lead-in fighter trainer. M-35582-7-U, 
C3195n, Research study, AUL/LRX, Maxwell AFB, AL 36112, 1972. 

Carter, G. W. Social indicators: A perspective of the social indica- 
tors movement. Paper Presented at the Annual Meeting of the 
Western Conference on the Uses of Mental Health Data, Newport 
Beach, California, October 1971. 

Carter, J. T. A new type of manual aviator. M-32983-U, C3232Na, 
Professional study no. 4046, AUL/LRX, Maxwell AFB, AL 36112, 
1970. 



12 



Cashj E. G. Educational opportunities in the air force. M-32983-U, 

C3383e, Professional study^ AUL/LRX, Maxwell AFB, AL 36112, 1972. 

Catalanelio, R. F. & Kirkpatrick, D, L. Evaluating training programs — 
the jtate of the art. Training and Development Jo urnal , 1968, 
22 (5), 2-9. 

Center fo^ Community Studies and the Nashville Urban Observatory. Social 
change and quality of life. A Report on a Conference, Nashville, 
Tennessee, May 7-9, 1970. 

Center for Social Policy and Community Development. Curricula and 

training techniques for the training of paraprof essional employees 
of public housing authorities. Philadelphia: Temple University, 
January 1972. 

Chafkin, S. H. et al. Measuring the impact of VISTA on poverty 

communities. Washington, D. C: American Technical Assistance 
Corp., January 1969. (ERIC ED 044 632.) 

Chaiklin, H. Evaluation research and the planning-programming-b /^geting 
system. In E* £. Schwartz (Ed.), Planning-programming-bud^ . ' ' ng 
systems and social welfare . Chicago: University of Chicago ^hool 
of Social Service Administration, 1970. 

Chambers, A. N. Development of a taxonomy of human performance: A 
heuristic model for the development of classification systems. 
Silver Spring, Maryland: American Institutes for Research, 1969. 
(ERIC ED 033 292.) 

Chant, R. J. A critical review of methods and techniques of instruction 
presently employed in the training of parachute packers. M-35582- 
7-U, C459c, Thesis no. 0185-68, AUL/LRX, Maxwell AFB, AL 36112, 1968. 

Chilson, J. S. Occupational qualifications and success of on-the-job 
trainees. Columbus: Ohio State University, 1969. (ERIC ED 045 
911.) 

CI. . ; C. C. The need for a new weapons training center in the European 
theater. M-35582-7-U, C5921n, Research study, AUL/LRX, Maxwell 
AFB, AL 36112, 1972. 



19G 



13 



Clark, ii, F. Classrooms In the military : An account of edu ation in 
ihjs firmed forces of tha United States , New York: Burc au of 
Publications, Teachers College, Columbia University, 1964. 

Clark, K. B. & Hopkins, J. A relevant war against poverty . New York: 
Harper & Row, 1969. 

Clark, W. E. Community power and decision-making: A selective biblio- 
graphy. Report no. CPL-ED-234, Monticello, Illinois: Council 
of Planning Librarians, 1971. (ERIC ED 058 518.) 

Clifton, 0. B. Methods of determining inservice training needs of 
beginning county extension agents. College Station, Texas: 
Texas A and M University, 1969. (ERIC ED 047 215.) 

Clinton, L., Stauss, J. H. & Chadwick, B. A. An evaluation of a federal 
program for improving the employability of emigrants from Indian 
reservations. Pullman, Washington: Washington State University, 
1972. 

Cochran, W. G. Sampling techniques . New York: John Wiley and Sons, 
1953. 

Cohen, M. D., March, J. G. & Olsen, J. P. A garbage can model of 

organiEational choice. Administrative Science Quarterly , 1972, 
17 (1), 1^25. 

Cohen, W. J. Education and learning. The Annals of the American Academy 
of Political and Social Science ^ 1967, 373 , 79-101. 

Coleman, F. E. A critique of the air force drug abuse education program. 

M-35582-7-U, C6922c, Research study, AUL/LRX, Maxwell AFB, AL 36112, 
1972. 

Coleman, J. S. Resources for social change : Race in the United States . 
New York: Wiley Interscience, 1971. 

Columbia University. Toward research on community action agencies: Key 
variables and research instruments* New York: Office of Economic 
Opportunity, Bure u of Applied Social Research, October 1966. 
(ERIC ED 058 353.) 



197 



14 



Concannon, M. R. & Grate, R. E. The development and test of a job 

inventory for a job analysis of the 64XX supply officer utilization 
field. SLSR-1-70, AFIT/L, Wright-Patterson AFB, Oil 4^ 33. 

Connally, J. W. A study of attrition in undergraduate pilot: training. 

M-35582-7-U, C7525s, Research study no. 0405-71, AUL/LRX, Maxwell 
AFB, AL 36112, 

Connecticut Department of Community Affairs. UCONN report first p 
in DCA training program. Action, July-August 1970. 

Cook, J. A. Quantity or. quality? An analysis r f current UPT phi:^ -ophy. 
AUL/LRX, Maxwell AFB, AL 36112, 1972. 

Corgell, K. S. Counseling and the leader, M-35582-7-U, C8331c, 

Research study no. 0330-70, AUL/LRX, Maxwell AFB, AL 36112, 1970. 

Cornblit, 0., DiTella, T. S., & Gallo, E. A model for political change 
in Latin America. Social Science Information , 1968, 7_, 13-48. 

Coroneos, P. P. Army officer procurement. M-35582-7-U , C797a, Thesis 
no. 0235-68, AUL/LRX, Maxwell AFB, AL 36112, 1968. 

Corrigan, R. E. Methods-means selection: An inquiry/decision-making 
process approach. Paper Presented at Planning Conference, 
California Polytechnic, Los Angeles, 1969. (ERIC ED 037 817.) 

Coster, J. K. & Morgan, R. L. The role of evaluation in the decision- 
making process. Center occasional paper no. 5, Raleigh, North 
Carolina University, Center for Occupational Education, 1969. 
(ERIC ED 042 028.) 

Crafton, T. J. An analysis of nursing education. M-35582-7, C885a, 
Thesis no. 0490-67, AUL/LRX, Maxwell AFB, AL 36112, 1967. 

Craig, R. L. & Bittel, L. R. (Eds.) Training and development hand- 
book . New York: McGraw-Hill, 1967. 

Crawford, T. M. The air borne forward air controller — peace time casualty. 
M-32983-U, C8991a, Professional study no. 3894, AUL/LRX, Maxwell 
AFB, AL 36112, 1970. 



EKLC 



198 



15 



Crim, S. C. Prescription for developing military leadership. 

M^35582^7-U, C929p, Thesis no. 0265-68, AUL/LRX, Maxwell AFB, 
AL 36112, 1968. 

Crist, J. A. Tactical fighter replacement training courses: An 
analysis. M-35582-7-U, C933C, Thesis no. 0500-67, AUL/LRX, 
Maxwell AFB, AL 36112, 1967. 

Croil, J. A. The Hannibal community survey: A case study in a community 
development technique. Columbia, Missouri: Missouri University, 
School of Social and Community Services, 1969. (ERIC ED 042 101.) 

Crum, J. M. & Turner, B. W. An analysis of the future role of f ill 

flight simulators in undergraduate pilot training. M-35582-7-U, 
C956a, Research study, AUL/LRX, Maxwell AFB, AL 36112, 1972. 

Currier, W. R. The citizen fighter pilot confirming an age-old heritage. 
M-35582-7-U, C976c, Research study no. 0360-70, AUL/LRX, Maxwell 
AFB, AL 36112, 1971. 

Cushman, J. H. & Townsend, J. N. An analysis of the educational re- 
quirements of a wing level chief of aircraft maintenance in the 
disciplines of production management, communicative skills, and 
statistics. SLSR-35-71A, AFIT/L, Wright-Patterson AFB, OH 45433. 

Dabbs, T. D. & Chadbourne, R. F. An analysis of the ALR-64830A supply 
systems computer specialist course. SLSR-8-67, AFIT/L, Wright- 
Patterson AFB, OH 45433, 1968. 

D*Agostino, R, B. Social indicators: A statistician's view. A Paper 
Presented in the Symposium on Social Indicators at the Annual 
Meeting of the American Psychological Association, Boston Univer- 
sity, September 1971. 

Dalglish, R. L. A report on adult education programs San Antonio Junior 
College district for the school year 1971-1972 and projected pro- 
grams for the school year 1972-1973. San Antonio, Texas: San 
Antonio College, A.dult Distributive Education, 1973. 

Daniel, D. M. Training strategy and the ^'system". Personnel Journal , 
1972, 15, 413-417. 



199 



16 



Darlington, R. B, & Stauffer, G. F. Use and evaluation of discrete 
test information in decision-making. Journal of Applied 
Psychology , 1966, 50, 125-129. 

Davidson, A. B, New concepts and problems in urban-inner city planning. 
Managerial Planning , 1972, 21 (2), 21-26. 

Davidson, F. P. Macro-engineering: A capability in search of a 
methodology. Futures , 1968, 1^, 153-161. 

Davis, C. H. Pilot resource procurement and management concept. 

M-32983-U, D2613p, Professional study, AUL/LRX, Maxwell AFB, 
AL 36112, 1972. 

Davis, S. L. Forward air controller continuation training. M-32983-U, 
D26Af, Research report no. 3594, AUL/LRX, Maxwell AFB, AL 36112, 
1968. 

Dayton, R. D. Airborne psychological warfare in South Vietnam. M-35582- 
7-S, D276a, Thesis no. 0300-68, AUL/LRX, Maxwell AFB, AL 36112, 
1968. 

Deal, J. E. The method of training USAF MAAG advisor personnel: Is 
is effective? M-35,582-7-U, D279m, Research study no. 0393-70, 
AUL/LRX, Maxwell AFB, AL 36112, 1971. 

Dean, M. E. An analysis of a human relations training program for 

squadron officer school section commanders. M-35582-7-U, D2821a, 
Thesis no. 0303-68, AUL/LRX, Maxwell AFB, AL 36112, 1968. 

Deane, D. H. & Hammond, K. Jl, Acquisition and applJ "ion of knowledge 
in complex inference tasks. Journal of Experimental l^^ sychology , 
1972, 92 (1), 20-26. 

Dearden, J. MIS is a mirage. Harvard and Business Rs^view ,' 1972, 50 , 
90-99. 

DeFrank, D. A. Thr impact of personnel training and technical data 
preparation on maintainability of a weapon system. (M. B. A. 
Thesis, University of Chicago), 1965. 



200 



17 



Demeratli, N. J. Trends ami anti-^treiulB In rellj^louH clwin^f. In K. U. 

Sheldon & W. Moort* (J'ids.), IndleuCorH of aoclnl riinn^c , Nrw Vt>rk: 
Russell Sage Foundation, 1968. 

Demock, H. G. Sensitivity training as a method of increasing on-the-job 
effectiveness. Sociological Inquiry , 1971, 4_1 (2), 227-232. 

DeNelsky, G. Y & McKee, M. G. Prediction of job performance from assess- 
ment reports: Use of a modified q-sort technique to expand pre- 
dictor and criterion variance. Journal of Applied Psychology , 
1969, 53, 439-445. 

DeNeufville, J. I. Social indicator design and use. Unpublished Doc- 
toral Dissertation, Massachusetts Institute of Technology, 1972. 

Deneva, C. Training evaluation causes in behavior. Personnel Administra- 
tion , 1969, 32 (5), 54-56. 

Denton, D. L. The application of economic analysis to the problem of 

aircrew training. M. B. A. Thesis, the University of Chicago, 1967. 

Department of the Air Force. A guide to the evaluation of civilian 
personnel management activities in the air force. Air Force 
Manual no. AFM40-5, October 1957. 

Department of the Air Force. Instructional system development. Air 

Force Manual no. 50-2, Washington, D. C: Department of the Air 
Force, Headquarters, U. S. Air Force, December 31, 1970c 

Department of the Air Force* Course charts, ate regulation 52-6. 

Randolph AFB, Texas: Headquarters Air Training Command, May 17, 
1971. 

Department of the Air Force. Kow to prepare ate training literature. 

ATC Manual no. 50-6, Randolph AFB, Texas: Air Training Command, 
August 31, 1971. 

Department of the Air Force. Patterns of technical training. ATC Pamphlet 
no. 52-1, Randolph AFB, Texas: Headquarters Air Training Command, 
December 20, 1971. 



291 

o 

ERIC 



Department of the Air Force. Occupational survey program. ATC 

Regulation no. 52-22, Randolph AFB, Texas: Headquarters Air 
Training Command, March 7, 1972. 

Department of the Air Force. Student critique program. ATC Regulation 
no. 52-29, Randolph AFB, Texas: Headquarters Ai.r Training Command 
March 7, 1972. 

Department of the Air Force. Training materials. ATC Regulation no. 
52-18, Randolph AFB, Texas: Headquarters Air Training Command, 
June 1, 1972. 

Department of the Air Force. Instructional system cevelopment. ATC 

no. 52-33, Randolph AFB, Texas: Headquarters Air Training Command 
July 3, 1972. 

Department of the Air Force. Course training standards. ATC Regulation 
no. 52-17, Randolph AFB, Texas: Headquarters Air Training Command 
August 29, 1972. 

Department of the Air Force. Training evaluation and course reviews. 
ATC Regulation no. 52-1, Randolph AFB, Texas: Headquarters Air 
Training Command, September 27, 1972. 

Department of the Air Force. Plans of instruction. ATC Regulation no. 
52-7, Randolph AFB, Texas: Headquarters Air Training Command, 
October 13, 1972. 

Department of the Air Force, the Army, and the Navy. Cost/schedule 

control systems criteria joint implementation guide. Washington 
D. C: the Secretary of Defense, March 1972. 

Department of Defense. Opportunities to increase effectiveness of long- 
term, full-time training program for civilian employees. Report 
to the Congress, June 1972. 

Department of Housing and Urban Development. Small community needs: A 
study of small community needs as related to federal housing and 
community development assistance. Washington, D. C, 1971. CERIC 
ED 063 987.) 



202 



19 



Departmeni: of Housing and Urban Development. Local strategies to affect 
state plan allocation of federal funds. Community Development 
Evaluation Series, Washington, D. C, 1972, 46_ (3). 

Department of Labor, Health, Education and Welfare. Opportunities for 
improving training results and efficiency at the East Bay Skills 
Center, Oakland, California, under the manpower development act. 
Report to Congress from Comptroller General of the United States, 
Washington, D. C., February 10, 1971. 

Department of the Navy. Evaluation study of the supply management core 
development program inventory control option. Philadelphia, 
(undated) 

Department of the Navy. Proceedings of symposium on evaluation of educa- 
tional technology applications. U. S. Naval Academy, June 22-26, 
1970. 

Dial, 0. E. Notes on the prospects for social indication. A Paper 

Presented at the Annual Meeting of the American Political Science 
Association, Los Angeles, California, September 1970. 

DiFilippo, ?• A. Pilot training for future needs. I4-32983-U, D569p, 
Research Report no. 3596, AUL/LRX, Maxwell AFB, AL 36112, 1968. 

Dillman, D. A. & Christenson, J. A. Towards the assessment of public 

valves. Unpublished manuscript, Pullman, Washington: Washington 
State University, Department of Sociology, 1971. 

Dluhy, M. Housing and social indicators. Unpublished manuscript, Ann 
Arbor, Michigan: Univer^sity of Michigan, School of Social Work, 
1969. 

Dodd, C. S. An analysis of the unit KC-135 flight trsTning process. 

M-35582~7-U, D639a, Research study no. 0545-71, AUL/LRX, Maxwell 
AFB, AL 36112 

Doty, R. E. A practical look at analysis and selected applications for 
air force planning. M~35582-7-U, D725p, Thesis no. 0340-68, 
AUL/LRX, Maxwell AFB, AL 36112, 1968. 

Dowling, W. D. A report of the educational component of a coupled on- 
the-job training program. Columbus, Ohio: Urban League and Ohio 
State University Center for Adult Education, August, 1970. (ERIC 
ED 045 878.) 

205 



ERIC 



20 



Downes, B. & Friedman, L, A. Local level decision-making and public 
poli<"-y outcomes: A theoretical perspective. In H. Hahn (Ed.), 
Urban affairs annual review . Vol. 6. P eople and politics in urban 
society ^ Beverly Hills: Sage Publications, 1972. 

Draft: joint federal/state task force on evaluation comprehensive eval- 
uation system — current status and development requirements. Wash- 
ington, D. C: Scientific Educational Systems, Inc., January, 1970. 
(ERIC ED 042 8l7.) 

Drescoll, D. L. Missile combat crew morale: Its impact on officer 
retention. M-35582-7-U, D7S12m, Research study, AUL/LRX, 
Maxwell AFB, AL 36112, 1972. 

Drewnowski-, J. Social and economic factors in development — introducLory 
considerations on their meaning, measurement and interdependence. 
Report no. 3, Geneva, Switzerland: United Nations Research 
Institute for Social Development, February, 1966. 

Drewnowski, J. Studies in the measurement of levels of living and 

welfare. Report no. 70.3, Geneva, Switzerland: United Nations 
Research Institute for Social Development, 1970. 

Dubin, R. Causality and social systems analysis. A Paper Presented at 
the Annual Meeting of the American Sociological Association, 
Denver, Colorado, August 30 — September 2, 1971. 

DuBois, P. H. & Mayj» G. D. (Eds.) Research strategies for evaluating 
training. Washington, D. C: American Educational Research 
Association, 1970. (ERIC ED 043 804.) 

Duke, E. A case study of management technique. Personnel Journal , 



Duncan, D. M. Training strategy and the "system." Personnel Journal , 
1972, June, 413^418. 

Duiican, 0. D. Social stratification and mobility: Problems in the 
measurement of trend. In E. B. Sheldon & W. E. Moore (Eds.), 
Indicators of social change ; Concepts and measurements . New 
York: Russell Sage Foundation, 1968. 



1972, 51 (1), 32-5. 



ERIC 




21 



Eckeikamp, V. C. J. Developing In-tllght judgment. M-33582-7-U, 
E191d, Research study, AUL/LRX, Maxwell AFB, AL 36112, 1972. 

Eckelkamn, V. r. j, T-.38 pilot instructor training. M-35582-7-U, 
Ei91t, Research study, AUL/LRX, Maxwell AFB, AL 36112, 1972. 

Educational Innovators Press. Performance and process objectives. 
Tucson, Arizona, 1970. (ERIC ED 054 606.) 

Educational Research Service. Evaluating adrainis trat ive /supervisory 

performance. Washington, D. C, November 1971. (ERIC ED 058 155.) 

Eisinger, P. K. People and politics in urban society. In H. Hahn (Ed.), 
Urban affairs annual review . Vol. 6. Beverly Hills: Sage 
Publication, 1972. 

Eitington, J. Film and group = more learners. Training in Busines s 
and Industry , 1972, ^ (1), 42-43. 

Eliot, B. F. Laterality, reaction-decision time, and pilot training 

effectiveness. Engineering Report (M. S.), Arizona State Univer- 
sity, 1969. 

Elmer, D. A. Revival of the air combat maneuvering training program. 
M-32983-U, E48r, Professional Study, AUL/LRX, Maxwell APR, AL 
36112, 1972. 

Engel, G. W. An analysis of the initial qualification training program 
for B-52 aircrews. M-35582-7-U, E571a, Thesis no. 0655-67, AUL/ 
LRX, Maxwell AFB, AL 36112, 1967. 

Eczioni, A. Indicators of the capacities for societal guidance. The 
Annals of the American Academy of Political and Social Science , 
1970, 388. 25-34. 

Etzioni, A. & Letiman, W. Some dangers in 'valid' social measurement. 

The Annals of the American Academy of Political and Socic.1 Science , 
1967, 373 , 1-15. 

Evans, J. W. Political factors in evaluation — the case of Head Start. 
Washington, D. C: U. S. Office of Education, 1971. 



20-j 



22 



Everett, R. P. & Hall, W. Regional training centers: A new venture in 
management development and intergovernmental cooperation. Sacra- 
mento, California: California State Personnel Board, March 1971. 

Extension Teaching and Field Service Bureau, Division of Extension. 
Information sheet. Austin: University of Texas at Austin, 
(undated) 

Farace, R. V. & Russell, H. W. Beyond bureaucracy — message diffusion 
as a communication audit tool . East Lansing: Michigan State 
University, April 1972. 

Ferriss, A. L. Indicators of trends in the status of American women . 
New York: Russell Sage Foundation, 1971. 

Fiedler, F. E. Predicting the effects of leadership training and ex- 
perience from the contingency model. Journal of Applied Psychology , 
1972, 56 (2), 114-19. 

Fincher, C. F. F-111 aircrew training for low altitude operations. 
M-35582-7-U, F4921f , Research study, AUL/LRX, Maxwell AFB, AL 
36112, 1972. 

Finstierbusch, K. Dimensions of nations: Inductively and deductively 
developed and propositionally related. A Paper Presented at the 
Conference on Methodological Problems in Compar'^tive Sociological 
Research of the Institute for Comparative Sociology, Bloomington, 
Indiana, April 1971. 

Fitzgibbon, T. J. Evaluation in the inner city. Paper Presented at the 
Invitational Conference on Measurement in Education, Philadelphia, 
Pennsylvania, April 1969. (ERIC ED 049 269.) 

Flanagan, J. C. & Miller, R. B. The performance record, administrator's 
manual. Chicago: Science Research Associates, 1955. 

Flanagan, J. C. - Miller, R. B. The performance record, handbook for 
foremen and supervisors of hourly employees. Chicago: Science 
Research Associates, 1955. 

Flanagan, J. C. & Miller, R. B. The performance record, handbook for 
superintendents and general foremen. Chicago: Science Research 
Associates, 1955. 

206 



23 



Flanagan, J, C, & Miller, R, B. The performance record, handbook 

for super'/lsor.s of nonsupervlsory salaried i=mployees. Chicago: 
Science Research Associates, 1955. 

Flanagan, J. C. & Miller, R, B. SRA catalog for business. Chicago: 
Science Research Associates, 1973. 

Flaten, E. A. An increasing requirement for the r. ofessional military 
education of our national guard officers. M-32983-U, F5863i, 
Professional study no. 4040, AUL/LRX, Maxwell AFB, AL 36112, 1970. 

Flax, M. J. Future prospects for the development of additional social 

and urban indicators. Working paper no. 1206-2, Washington, D. C. 
The Urban Institute, 1971. 

Fleishman, E. A. (Ed.) Studies in personnel and industrial psychology ^ 
Homewood, Illinois: Dorsey Press, 1961. 

Fleishman, E. A. Attitude vs. skill factors in work group productivity. 
Personnel Psychology ^ 1965, 18, 253-266. 

Fleishman, E. A. On the relation between abilities, learning, and human 
performance. ^Imerican Psychologist , 1972, _27 (11), 1017-1032. 

Flynn, R. R. Electronic warfare officer training. M-35582-7-U, F648e, 
Research study no. 0675-71, AUL/LRX, Maxwell AFB, AL 36112, 1971. 

Folk, M. A critical look at the cross impact matrix method. A Research 
Report, Washington, D. C: Syracuse University Research Corp. for 
the National Center for Educational Research and Development, 
August 1971. (ERIC ED 061 635.) 

Ford, J. B. Contextual content analysis: A link between micro-social 

and macro-social research. A Paper Presented at the Annual Meeting 
of the American Sociological Association, Denver, Colorado, 
August 30~September 2, 1971. 

Forsyth, R. A. Considerations to the usefulness of the performance indi- 
cators in Dyer's student change model of an educational system. 
Paper Presented at Annual Meeting of American Educational Research 
Association, Chicago, April 1972. (ERIC ED 063 332.) 



207' 



24 



Fortune, J. C. Problems facing the evaluator of social action programs • 
A Paper Presented at the Symposium on Evaluation Prospective^-: A 
Focus on Current Gaps at the Annual Meeting of the American Educa- 
tional Research Association, Chicago, April 1972. (ERIC 064 380.) 

Fosse, J. M. Technical training — the key to a true volunteer air force, 
M-35582-7-U, F752t, Research study, AUL/LRX, Maxwell AFB, AL 36112, 
1972. 

Prey, E. P. Spin training: A treatise. M-35582-7-U, F8931s, Research 
study, AUL/LRX, Maxwell AFB, AL 36112, 1972. 

Froberg, S. E. A guide for the development of an inservice education 
program. Tampa: Florida Regional Medical Program, May 1971. 
(ERIC ED 049 431.) 

Fulmer, R. M. The future of training for the future. Training in 
Business and Industry , 1972, February, 40-41. 

Gable, S. R. Laboratory training — its role in air force manager 

development. M-35582-7-U, G1151, Thesis no. 0805-67, AUL/LRX, 
Maxwell AFB, AL 36112, 1967. 

Gagne, R. M. (Ed.) Psychological principles in sysc • development . 
New York: Holt, Rinehart & Winston, 1962. 

Gail, J. W. An incentive program for air reserve forces personnel: 

College tuition assistance, skill knowledge training, and promotion. 
M-32983-U, 6138i, Professional study, AUL/LRX, Maxwell AFB, AL 
36112, 1972. 

Galnoor, I. Social information for what? The Annals of the American 
Academy of Political and Social Science , 1971, 393 , 1-19. 

Garn, H. A. & plax, M. J. Urban Institute indicator program. Working 
Paper 1206-1, Washington, D. C: The Urban Institute, July 1971. 

Garstka, J. A. A comparative analysis of training reports. M-35582-7-U, 
G243c, Thesis no. 0820-67, AUL/LRX, Maxwell AFB, AL 36112, 1967. 

Gastil, R. D. Social indicators and quality of life. Public Administra- 
tion Review , 1970, 30., 596-601. 



o . 20 



ERIC 



S 



25 



Gentry, J, C. Job related education as a rocruitment/retention device 
in the all volunteer force. M-32983-U, G339j, Professional 
study, AUL/LRX, Maxwell AFB, AL 36112, 1972. 

Gerbner, G. Cultural indicators: The case of violence in television 

drama. The Annals of the American Academy of Political and Social 
Sc ience , 1970, 388 , 69-81. 

Germann, D. B, Management decision education for the young officer. 
M-35582-7-U, G373m, Thesis no. 0840-67, AUL/LRX, Maxwell AFB, 
AL 36112, 1967. 

Giammateo, M. G. Target group analysis: Aids for evaluation. Field 

Paper 9, Contract OEC-4-7-062871-3054 , Portland, Oregon: Northwest 
Regional Educational Lab, April 1969, (ERIC ED 031 785,) 

Giguere, A, Aircrew standardization — an analysis. M-35582-7~U, 

G4615, Thesis no. 0860-67, AUL/LRX, Maxwell AFB, AL 36112, 1967. 

Gilroy, K. A. Electronic warfare training for tactical fighter aircrew 
(U). M-35582-7-S, G489e, Research study no. 0540-70, AUL/LRX, 
Maxwell AFB, AL 36112, 1971. 

Gitter, A. G. Factor analytical approach to indexing multivariate data. 
Report no, 43, Boston: Boston University, Communication Research 
Center, 1970. 

Gitter, A. G. & Knoche, E. Importance ratings of sixteen aspects of 
life. Report no. 59, Boston: Boston University, Communication 
Research Center, 1971. 

Glaser, R. Training research and education . Pittsburgh: University of 
Pittsburgh Press, 1962. 

Glaser, R. Instructional technology and the measurement of learning 
outcomes. American Psychologist , 1963, 1S_ (8), 519-521. 

Glaser, R. & Glanzer, M. Abstract of training and training research. 
Report no. Nonr-2551-00 , Pittsburgh: American Institutes for 
Research, August 1958. (ERIC ED 044 578.) 



209 



26 



Glass, G. J. Course design and media selection for tiaining C-141 
pilots. M-35582-7-U, G549c, Research study no. 0545-70, AUt./ 
LRX, Maxwell AFB, AL 36112, 1971. 

Goetz, C. T. Outline for air force teacher training manual — Protestant. 
358.34, A2988S, Student Report no. 9, AUL/LRX, Maxwell AFB, AL 
36112, 1960, 

Gold, B. K. Evaluation of programs. Paper Presented at a Conference 
Sponsored by the Compensatory Education Project, Coordinating 
Board, Texas College and University System, Austin, Texas, April 
5-6, 1971. (ERIC ED 048 845.) 

Goldwasser, E. L. Science and man: Breaking new ground at Batavia. 
Bulletin of the Atomic Scientists , 1969, 25^ (8), 7-10. 

Cooler, D. D. Data collection for educational decision-making: Estab- 
lishing priorities. Report no. WP-No-1, Urbana: Illinois 
University, Center for Instructional Research and Curriculum 
Evaluation. (undated) (ERIC ED 046 998.) 

Cooler, D. D. & Grotelueschen , A. D. Curriculum development accountabil- 
ity. The Education Digest , 1972, January, 21-:3. 

Gordon, A. C., Campbell, D. T., et al. Recommended accounting proce- 
dures for the evaluation of improvements in the delivery of state 
social services. Preliminary draft, Evanston, Illinois: North- 
western University, Center for Urban Affairs, 1970. 

Gordon, T. J. & Ament, R. H. Fo'-'casts of some technological and scien- 
tific developments and societal consequences. Middletown, 
Connecticut: Riverview er. Institute for the Future, 1969. 

Gottschalk, L. A. & Gleser, G. v>. The measurement of psychological 
states through the content analysis of verbal behaviors . Los 
Angeles: University of California Press, 1969. 

Grassman, J. W. Why not an aviation cadet program? M-35582-7-U, G 

;69w. Research study no. 0560-70, AUL/LRX, Maxwell AFB, AL 36112, 
1970. 

Gray, K. E. Competencies needed by personnel engaged in program planning 
in state divisions of vocational-technical education. 1970. 
(ERIC ED 042 927.) 



2U 



27 



Greve, J. Voluntary housing in Scandinavia. Publi c Administration 

Journal of the Royal Institute of Public Administration , London, 
1972, 50 (Summer). 

Grobman, H. Developmental curriculum projects ; Decision points and 
p rocesses . New York: F. E. Peacock, 1970. 

Gronlund, N. E. Stating behavioral objectives for classroom instruction . 
New York: The Macmillan Co., 1970. 

Gross, B. M. The coming general systems models of social systems. Human 
Relations , 1967, 2£ 357-374. 

Gross, J. R. The air force officer evaluation program: A comparison of 
civilian industry and the air force, M-35582-7-U, G878a, Thesis 
no. 0940^67, AUL/LRX, Maxwell AFB, AL 36112, 1967. 

Grove, W. T. Analysis of unit level combat readiness training. M-35583- 
7-U, G883a, Research study, AUL/LRX, Maxwell AFB, AL 36112, 1972. 

Gruenfeld, L. Ter.'ionality needs and expected benefits from a manage- 
ment development progra^^. Occupational Psychology , 1966, 40 
(1&2), 75. 

Guba, E, G. Evaluacion as a decision-making tool. Speech Given at 

Audio-Visual Conference, Indiana University, June 1970. (ERIC ED 
055 362.) 

Kaas, J. A^ Middle managers' expectations of the future world of work: 
Implications for management development. Pittsburgh: Pittsburgh 
University, 1969. (ERIC ED 041 207.) 

Haggard, D. F. et al. An experimental program of instruction on the 

management of training. Report no. IIumRRO-TR-70-9, Alexandria, Vir- 
ginia: Human Resources Research Organization, June 1970, (ERIC ED 
047 195J 

Kaggerty, S. K. Military subsidies to the American society. M-32983-U, 
H1452m, Professional study no. 3921, AUL/LRX, Maxwell AFB, AL 
36112, 1970. 

Kahn, A. J. Community decision-making systems. November 1970. (ERIC 
ED 05' ^^06.) 



211 



28 



Hamlin, D. A. An analysis of executive development in the U. S. 

Air Force. M-35582-7-U, H223a, Thesis no. 0475-68, AUL/LRX, 
Maxwell AFB, AL 36112, 1968. 

Hampton, L. A. An analysis of critical behaviors of cooperative exten- 
sion agents in the performance of their jobs. Ed.D Thesis, 
Raleigh: North Carolina State University, 1969. (ERIC ED 042 
978.) 

Hansen, M. H. et al. Sample survey methods and theory. New York: 
John Wiley & Sons, 1953. 

Harmon, P. Developing performance objectives in job training progrrams. 
Educational Technology , 1968, November, 11-16. 

Harris, E. C, Sensitivity training and managerial development., M- 32983- 
U, H313s, Professional study no. 4042, AUL/LRX, Kax'vv^ell AFB, AL 
36112, 1970. 

Harsanyi, C. A. & White, C. E. An analysis of the effectiveness of the 
weapons mechanic training program. SLSR-10-67, AFIT/L, Wright- 
Patterson AFB, OH 45433, 1968. 

Hartke, R. H. Effectiveness in cost effectiveness araiysis. M-35582- 

7-U, :329e. Thesis no. 0500-68, AUL/LRX, Maxwell AFB, AL 36112, 1968. 

Hatch, J. C. Television performance effectiveness: A study of related 
variables and the effects of inservice training and evaluative 
feedback. Ph.D. Thesis, Madison: The University of Wisconsin, 
1968. (ERIC ED 045 913.) 

Hauser, P. M. The chaotic society: Product of the social morphological 
revolution. American Sociological Review , 1969, 14, 1-19. 

Hauser , P.M. Political implications of social indicators. A Paper 

Presented at the Annual Meeting of the American Political Science 
Association, Chicago, September 1971. 

Hazel, J. T. Reliability of job ratings as a function of time spent on 
evaluation. Journal of Industrial Psychology , 1966, 16-19. 

Heaphy, J. F. Community relations training: An alternative view. Journal 

of Criminal Law , Criminology and Police Science , 1971, 62^ (4), 570-572. 



212 



iieiler, F. A. <S Porter, L. W, Perceptions of managerial needs and skills 

in two national samples. Occupational Psychology ^ 1966, 40 (122), 1. 



Helton, C. T. The need for an expanded reading program. M-35582<-7~U, 
M4841n, Research study, AUL/LRX, Maxwell AFB, AL 36112, 1972. 

Henriot, P. J. Political questions about social indicators. The Western 
Political Science Quarterly , 1970, 23_> 235-255. 

Heynan, M. M. Criteria and guidelines for the evaluation of in-service 

training. Washington, D. C.: Department of Health, Education, and 
Welfare, 1968. (ERIC ED 031 643.) 

Henrichs, J, R. Evaluation of an industrial training program. Proceed- 

ings of the Annual Convention of the American Psyciiological Associ a- 
tion , 1972, 2» 445-446. 

Hock, M. D. Considerations of decision theory in the reconstruction of 

logic in urban planning. Columbus, Ohio: Ohio State University, 1968 
(ERIC ED 034 299.) 

Hodgkinson, H. L. & Thelin, J. Survey of the applications and uses of 

unobtrusive measures in fields of social service. Berkeley: Univer- 
sity of California, August 1971. (ERIC ED 05S 863.) 

Hodgson, W, E. Some basic design considerations for army helicopter syn- 
fzhetic flight trainers. M-35582- "--U, H691s, Thesis no. 0550-68, 
AUL/LRX, Maxwell AFB, AL 36112, 1968. 

Holder, J. J. Evaluation of an in-company management training program. 
Training and Development Journal , April 1972. 

Holiday, R. C. Automated scheduling and record keeping for ADC ground 

environment training. M-32983-U^ H7321a, Professional stody, AUL/ 
LRX, Maxwell AFB, AL 36112, 1972. 

Hooker, C. P. & Mueller, V. D. A plan for performance review and compensa- 
tion of administrative personnel. Richfield, Minnesota: Richfield 
Independent School District 280, May 1969. (ERIC ED 043 094.) 

Hoos, I. R. Information systems and public planning. Management Science , 
1971, 17, B-658-B--671. 



213 



30 



Hopper, T. E. Air force advanced education: A cost-effectiveness approach 
M-32983-U, H7981a, Professional study no. 3387, AUL/LRX, Maxwell AFB, 
AL 36112, 1967. 

Hormay, A. L. Principles of rest-rotation grazing and multiple-use land 
management . Training test no. 4(2200), Washington, D. C: Forest 
Service U. S. Department of Agriculture, September 1970. 

Horowitz, I. L. Professing sociology ; Studies in the life cycle of social 
science . Chicago: Aldine Publishing Co., 1968. 

Horowitz, I. L. Social indicators and social policy. In I. L. Horowitz 
(Ed . ) , Professing sociolo . ^ ^y; Studies in the life cycle of social 
science . Chicago: Aldine Publishing Co., 1968. 

Houk, C. P. Formal training requirements for air force maintenance engineer- 
ing duties. M-35582-7-U, H8383f , Research study no. 0700-70, AUL/ 
LRX, Maxwell AFB, AL 36112, 1971. 

Housing in the inner city. Urban Research News , 1970, 4_ (15), 8. 

Hozel, J. T. Reliability of job ratings as a function of time spent on 
evaluation. Journal of Industrial Psychology , 1966, 4_, 16-19. 

Huggins, L. E. Modernization of the tactical fighter air-to-ground 

gunnery range. M-35582-7-U, H8913m, Research study, AUL/LRX, Maxwell 
AFB, AL 36112, 1972. 

Hutchinson, T. E. Some overlooked implications of the purpose: To provide 
data for decision-making. Presented at Annual Meeting of the 
American Educational Research Association, Chicago, April 1972. 

Hyder, C. M. Model cities training program. Evaluation Report, Chat- 
tar ..oga, Tennessee: Tennessee University, 1969. (ERIC ED 043 848.) 

The Illinois Department of Local Government Affairs. Intern plan to 

increase manpowt-r. Local Government News. September-October 1970. 

Illinois State Board of Higher Education. Public administration. Report 
of Master Plan Committee T, Springfield, Illinois, 1969. (ERIC ED 
040 650- ) 



ERIC 



31 



Institut de Recherche Economique et de Planif icat,ion. Recherche sur 
1 * integration desindicateurs .... Unpublished manuscript, 
Grenoble, France: University of the Social Sciences of Grenoble, 
1971. 

Institut National de la Statistique et des Etudes Economiques. Inven- 
taire bibliographique . Unpublished manuscript no. 320/1163, 
Prepared by the Social Statistics Group, Paris, France, 1971. 

Institut National de la Statistique et des Etudes Economiques. Presenta- 
tion des travaux Francais. Unpublished manuscript, Paris France: 
Comissariat General du Plan d'Equipment et de la Productivite , 1971 

Institut National de la Statistique et des Etudes Economiques. Social 

indicators. Unpublished manuscript, Paris, France: Social Statis- 
tics Group, 1971. 

Institute for Urban Studies. Housing management training curriculum. 

Houston, Texas: The University of Houston, November 1-13, 1971. 

Institute for Urban Studies. Housing management training — seminars and 
evaluation surveys. Houston, Texas: The University of Houston, 
November 1-13, 1971. 

Institute of Urba. and Regional Development. Advice as a socially con- 
structed activity. Working Paper no. 144C, Berkeley, California: 
University of California, 1971. 

Institute of Uroan Studies. Executive manpower for urban government . 
Vol. 2 Education for Career service. Arlington, Texas: The 
University of Texas at Arlington. 

International Association of Chiefs of Police, Professional Standards 
Divisi.on. Police reference notebook, section II (supervision 
index). 1971. 

Jackson, E. N. A factor analysis of small community development. Un- 
published Masters Thesis, Manhattan, Kansas: Kansas State Universit 
1970. 

Jaffee, C. L. Effective management selection: Analysis of behavior by 
simulation techniques. Reading, Massachusetts, 1971. (ERIC ED 
058 531.) 



215 



32 



Jamieson, G. H. A pilot survey of recruitment and training. Occupa - 
tional Psychology , 1966, 40 (1&2) , 167. 

Jew, B. More bang for the buck. M-32983~U, J59in, Professional study 
no. 4041, AUL/LRX, Maxwell AFB, AL 36112, 1970. 

Jew, B. Primary cause: Pilot factor. M-32983-U, J59p, Professional 
study no. 3^^38, AUL/LRX, Maxwell AFB, AL 36112, 1970. 

Johns, J. P. & Glo'^insky, A. Paraprof essional training model. A 
process for training. Contract no. OEG-0-9-324150--2235-725, 
Detroit, Michigan: Wayne County Intermediate School District, July 
1970. (ERIC ED 049 183.) 

Johns, J. P. & Glovinsky, A. Taxonomy of paraprof essional training. 

Contract no. OEG-0-9-324150-2235-725, Detroit, Michigan: Wayne 
County Intermediate School District, September 1970. (ERIC ED 
049 184.) 

Johnson, A. L. An analysis of on-the-job proficiency training. M-35582- 

7-U, J663a, Thesis no. 1210-67, AUL/LRX, Maxwell AFB, AL 36112, 1967. 

Johnson, N. J. & Ward, E. J. A new approach to citizen participation: 
An exploration tying information and utilization. Pittsburgh: 
Carnegie-Mellon University, School of Urban and Public Affairs, 1970. 

Johnsten, R. A. Motivation in mipsile training. M-35582-7-U, J72m, 
Research study, AUL/LRX, Maxwell AFB, AL 36112, 1972. 

Johnston, D. F. Forecasting methods in the social sciences. Technologi- 
cal Forecasting and Social Change , 1970, 2^, 173-187. 

Jones, W. G. List of publications : 1966-1970 . Ann Arbor, Michigan: The 
University of Michigan, Institute for Social Research, February 1971. 

Judge, A. J. Information systems and inter-organizational space. The 
American Academy of Political and Social Science , 1971, 393 , 
47-64. 

Justiss, B. R. The role of the flight simulator in transition training 
for jet transport pilots. M-35582-7-U, J96r, Research study, AUL/ 
LRX, Maxwell AFB, AL 36112, 1972. 



21G • 



33 



Kamrany, N. M. A note on the development of social indicators. Santa 
Monica, California: Systems Development Corporation, 1968. 

Karalus, P. H. An analysis of the USAF pilot training program. M-32983- 
U, KlSla, Professional study no. 4162, AUL/LRX, Maxwell AFB, AL 
36112. 

Katagiri, G. Oregon's pilot state dissemination program. Oregon State 
Board of Control, Contract no. OEC-0-70-4755 , Washington, D. C: 
National Center for Educational Communication, January 1972. 

Kaufman, R. A. Educational system planning . Englewood Cliffs, New 
Jersey: Prentice-Hall, Inc., 1972. 

Kaufman, R. A., Corrigan, R. E., & Nummelly, C. L. The instructional 

system approach to training. Human Factors , 1966, 9^ (2), 157-162. 

Kavanagh, M. J., MacKinney, A. C, & Wolins, L. Issues in managerial 
performance: Multitrait — multimethod analyses of ratings. 
Psychological Bulletin , 1970, 75^ (1), 34-49. 

Kellam, S. G. , Agrawal, K. C, Branch, J. D., & Grabill, M. E. Strategies 
in urban community mental health. Chicago: The University of 
Chicago, VJoodlawn Mental Health Center, 1972. 

Kelly, E. F. & Bunda, M. A. The development of a survey instrument for 

evaluative priorities: A field test. Paper Presented at the Annual 
Meeting of the American Educational R( search Association, New York, 
February 1971. (ERIC ED 049 317.) 

Kepner, C. H. & Tregoe, B. o. The rational manager . New York: McGraw- 
Hill, 1965. 

Kind, S. M. Gedrag van de consulent nader bekeken. [Activity of the out- 
side expert reviewed more closely]. Mens en Onderneming , 1969, 23 
(1), 27-37. 

Kipnis, D. & Glickman, A. S. The prediction of job performance. Journal 
of Applied Psychology , 1962, 46, 50-56. 

Kirkpatrick, D. L. Supervisory training and development. Madison, Wis- 
consin: American Society for Training and Development, 1971. 
(ERIC ED 054 450.) 



EKLC 



217 



34 



Kish, L. Survey sampling . New York: John Wiley & Sons, 1967. 

KiLchell, D. Training USAF MAAG advisor personnel. M-35582-7-U, K621t, 
Research study no. 1120-71, AUL/LRX, Maxwell AFB, AL 36112. 

Kline, R. W. Classroom leadership in professional military education. 
M-35582-7-U, K653c, Thesis no. 0673-68, AUL/LRX, Maxwell AFB, AL 
36112, 1968. 

Kohn, V. A selected bibliography of evaluation of management training 
and development programs. Hamilton, New York: American Founda- 
tion for Management Research, Inc., 1969. (ERIC ED 034 947.) 

Kohn, V. & Parker, T. C. Some guidelines for evaluating management 
development seminars. Training and Development Journal , 1969, 
23, 19--23. 

Kondrasuk, J. N. Selling: The name of the game in training. Training 
and Development Journal , 1972, 26. (6), 14-17. 

Konuitz, M. Civil liberties. The Annals of the American Academy of 

Political and Social Science , 1967, 371 , 38-58. 

Krathwohl, D. R. , Bloom, B. S., & Masio, B. B. Taxonomy of educational 
objectives ; Affective domain . New YorV: David McKay Co., Inc., 
1964. 

Krendel, E. S. The life cycle as a basis for social policy and social 

indicators. Working paper no. 106, Berkeley, California: Univer- 
sity of California, Institute of Urban and Regional Development, 1969. 

Krendel, E. S. A case study of citizen complaints as social indicators. 
IEEE Transactions on Systems Science and Cybernetics , 1970, SSC-6 
(4), 265-272. 

Krendel, E. S. Social indicators and urban systems dynamics. Socio- 
Economic Planning Sciences , 1971, 5^ (4), 387-393. 

Krieger, M. H. Planning an affect based society: Prediction, indicators 

and structure. Working paper 144B, Berkeley, California: University 
of California, Institute of Urban and Regional Development. 



218 



Kuhiman, A. J. An analysis of the feasibility of consolicis ting radar 

training and maintenance on the F/FR-4C/D aircraft. SLSR-50-71B, 
AFIT/L, Wright-Patterson AFB, OH 45433, 1972. 

Lanale, H. H. Changes in concepts of income adequacy over the last century. 
The American Economic Review , 1958, 47, 291-299. 

Lamson, R. W. Framework of categories for science policy analysis and 

technology assessment. Washington, D. C. : National Science Founda- 
tion, Office of Planning and Policy Studies, 1969. 

Land, K. C. Social inaicators. In R. B. Smith (Ed.), Social science 
methods . New York: The Free Press, 1970. 

Land, K. C. On the definition of social indicators. The American Sociolo - 
gist , 1971, 6^, 322-325. 

Lanier, S. L. & Swarts, R. F. Analysis of the management education require- 
ments for the base level chief of aircraft maintenance, SLSR-9-69, AFIT/L, 
Wright-Patterson AFB, OH 45433, 1970. 

Larche^ L. E. An analysis of team teaching in secular a id religious educa- 
tion. M-35582-7-U, L31^Ja, Thesis no. 1375-67, AUL/i.RX, Maxwell AFB, 
AL 36112, 1967. 

Layne, E. C. An analysis of disabling injuries incurred in military air- 
borne Training. M. S. Thesis, Georgia Institute of Technology, 1966. 

Leadership Resources, Inc. HUD/HAA training program. Washington, D. C, 
December 1968. 

Leadership Resources, Inc. Training outline and guide on: Professional 

management in public housing. Washington, D. , November 1970. 

Leadley, S. M. Working papers on rural community services. Washington, 
D. C: Department of Agriculture, 1972. (ERIC ED 062 064.) 

Lehman, E. W. Social indicators and social problems. In E. 0. Smigel 
(Ed. ) , Handbook on the study of social problems . Chicago: Rand 
McNally & Co., 1971. 

Lester, R. I. Criteria for evaluating training materials. Training and 
Dev e lopment Journal , 1971, _25 (8), 12-15. 



219 



36 



Levin, G. & Stein, D, D. System intervention In a school-community 
conflict. Journal of Applied Behavioral Science , 1970, 6^ (3), 
337-352. 

Levine, R. A, & Williams, A. P. Making evaluation effective: A guide. 

Pittsburgh: Carnegie Mellon University, May 1971. (ERIC ED 058 612.) 

Line, R. N. & Gordon, W. B. A budget manual for Texas cities and towns. 
Fort Worth, Texas, 1971. 

Lineberry, R. L. & Fowler, E. P. Reformism and public policies in American 
cities. American Political Science Review , 1967, 61^ (3), 701-716. 

Linowes, D. Accounting for social progress; Yardsticks must be found for 
public programs. (Extension of remarks of the Hon. F. E. Moss) Con - 
gressional Rec ord, 197". March, E2369-E2370. 

Liv, W. T. & Duff, R. W. The structural effect of communication flows in 
a pre-industrial city. A Paper Presented at the Annual Meeting of 
the Ohio Valley Sociological Society, Cleveland, Ohio, April 1971, 

Lloyd, E. A. Consolidated undergraduate flying training. M-32983-U, 
L793c, Professional study, AUL/LRX, Maxwell AFB, AL 36112, 1972. 

Long, N. E. Indicators of change in political institutions. The Annals of 
the American Academy of Political and Social Science , 1970, 388 , 
35-45. 

Lord, T. F. A report on a feasibility determination . Houston: Institute 
for Urban Studies, University of Houston, October 1971. 

Loupe, M. J. Training and transfer of problem-solving. Minneapolis: 
Minnesota University, April 1972. (ERIC ED 061 547.) 

Low-income housing study. Urban Research News, 1970, _4 (26) . 

Lowe, H., Adrian, R. 0 Brocklesly, I. A factor analytic study of selection 
decision-making. Occupational Psychology , 1968, 4^ (1). 

Luhrsen, D. E. An investigation of the benefits of certain air force training 
and research projects of the civilian economy and community. M-35582- 
7-U, L951i, Research study, AUL/LRX, Maxwell AFB, AL 36112, 1972. 



220 



37 



Lund, M. A guide to training of office supervisors. London: Britisli 
Association for Commercial and Industrial Education, November 
1969. (ERIC ED 0A2 964.) 

Lundgren, D. C. Trainer style and patterns of group development. The 
Journal of Applied Behavioral Sc ience , 1971, 2 (6), 689-709. 

Lupes, E. C. Correction of correlation with a dichotomus variable for^ 
restriction in range . Technical Report PRL-TDR-62-64 , Lackland 
AFB, Texasi 6570th Personnel Research Lab, Aerospace Medical Div. , 
Air Force Systems Command, December 1962. 

Luty, E. T. o Metos, T. H. Management through administrative differentia- 
tion: An operational strategy utilizing environmental inputs for 
viable action outputs. 1971. (ERIC ED 057 431.) 

hacArgel, R. E. An approach to laboratory training in squadron officer 
school. M-35582-7-U, M1151a, Thesis no. 0733-68, AUL/LRX, Maxwell 
AFB, AL 36112, 1968. 

Madden, J. M. The effect of varying the degree of rater familiarity in 
job evaluation. Personnel Administration , 1962, ^25 (6), 42-46. 

Madden, J. M. A further note on the familiarity effect in job evaluation. 
Personnel Administration, 1963, 26_ (6), 52-53. 

Madden, J. P. Social change and public policy in rural America: Data and 
research needs for the 1970's. American Journal of Agricultural 
Economics , 1970, 5_2, 308-314. 

Mager, R. F. Preparing instructional obj ectives . San Francisco: Fearon 
Publishers, 1962. 

Mahoney, T. A., Sorenson, W. W. , Jerdee, T. H., 6< Nash, A. N. Identification 
and prediction of managerial effectiveness. Personnel Administration , 
1963, _26, 12-22. 

Maier, N. R. Innovation in education. American Psychologist , 1971, 26_ 
(8), 722-725. 

Maier, N. R. Effects of training on decision-making. Psychological Reports , 
1972, 30 (1), 159-164. 



38 



Maimon, Z. Second-order consequences — a presentation of a concept. 

Detroit: Center for Urban Studies, Wayne State University, 1971. 

Main, E. D. A nationwide evaluation of M. D. T. A. institutional job 
training programs . Chicago: National Opinion Research Center, 
University of Chicago, October 1966. 

Main, E. D. A nationwide evaluation of M. D. T. A. institutional job 
training. Journal of Human Resources , 1968, 2^ (2). 

Malkemes, K. 0. Air force aero clubs — a recreational liability? M-35582- 
7-U, M251a, Thesis no. 1500-67, AUL/LRX, Maxwell AFB, AL 36112, 1967. 

Manpower Administration (DOL) . Toward the ideal journeyman. Vol. 1, An 

optimum training system in apprenticeable occupations. Manpower Re- 
search Monograph no. 20, Washington, D. C, 1970. (ERIC ED 047 112.) 

Mansfield, W. B. Job-related activities of first-line supervisors in Texas 
manufacturing industries. An Abstract of a Doctoral Dissertation Pre- 
pared by Author for Engineering Extension Service, Texas A and M 
University, Supervisory Development Division, 1970. 

Marcuse, P. Social indicators and housing policy. Urban Affairs Quarterly , 
1971, 2 (December), 193-217. 

Margolis, F. H. Training by objectives: A participant-oriented approach. 

Washington, D. C: Sterling Institute, June 1970. (ERIC ED 047 234.) 

Market Dynamics, Inc. The public's attitude toward public library services 

in Essex County, New Jersey. A Research Study Conducted for the 

Essex County Library Directors Group Public Relations Committee, 

Trenton: Kew Jersey State Library, June 1969. (ERIC ED 061 974.) 

Markley, 0. W. Alternative futures: Contexts in which social indicators 
must work. Research note. Report no. EPRC-6747-11 , Menlo Park, 
California: Stanford Research Institute, Educational Policy Center, 
February 1971. (ERIC ED 049 150.) 

Marks, D. M. Air attache: Realistic selection and meaningful training. 
M-32983-C, M3461a, Research Report no. 3641, AUL/LRX, Maxwell AFB, 
AL 36112, 1968. 



000 

^ ^ ^ 



39 



Marsh, P. FightergaLor — does he come prepared? M-35582-7-U, M366f, 
Research study, AUL/LRX, Maxwell AFB, AL 36112, 1972, 

Martin, M, E. The current population survey as a statistical information 
system. Washington, D. C: U, S, Office of Management and 
Budget, 1971, 

Martin, T, & Berry, K, J. Generalizing to individuals: Another problem 
of evidence and inference, A Paper Presented at the Annual Meeting 
of the Rural Sociological Association, Denver, Colorado, August 1971, 

Martinson, D, J, The selection criteria for air force ROTC instructors, 

M-35582-7-U, M386ls, Research study, AUL/LRX, Maxwell AFB, AL 36112. 

Maslow, A. P, Job factors, attitudes and preferences affecting the rela- 
tive advancement and turnover of men and women in federal careers, 
Washington, D, C: American Psychological Association, September 
1970. (ERIC ED 043 045,) 

Maxwell, H, L, Analysis of AFROTC and SOS curriculums, M-35582-7-U, 

M4652a, Thesis no. 1540-67, AUL/LRX, Maxwell AFB, AL 36112, 1967. 

McClure, E. E. & Groetz, R, E. Graduate curriculum developiu at for social 
planning specialization in urban and regional planning. Final re- 
port. Contract no, OEG-2-7-070008-2958 , Tallahassee, Florida: Florida 
State University, October 1968, (ERIC ED C26 981,) 

McConnell, J, J, An assessment center program for multi-organizational use. 
Training and Development Journal , 1972, March, 6-14. 

McDevitt, M. & McGowan, T. New York State's central social environment 
study: Social research for comprehensive planning. Albany, New 
New York: New York State Office of Planning Coordination, 1970. 

McDonald, J, M. The adequacy of army management training. M-35582-7-U, 
M1355a, Thesis no. 0800-68, AUL/LRX, Maxwell AFB, AL 36112, 1968. 

McGranahan, D. V, Analysis of socio-economic development through a system 

of indicators. Annals of the^ American Academy of Political and Social 
Science, 1971, 393, 65-81. 

McGranahan, D, Development indicators and development models. The Journal 
of Development Studies , 1972, 



223 



40 



McGuire, T. E. Cost analysis what is it and what should it be? M-35582- 
7-U, M1484c, Research study no. 0935-70, AUL/LRX, Maxwell AFB, AL 
36112, 1970. 

McHale, J. Global ecology: Toward the planetary society. The American 
Behavioral Scientist , 1968, 11, 29-33. 

Mcintosh, W. A. Social indicators and social change: An interpretive 
essay. Unpublished paper, Ames, Iowa: Iowa State University, 
Departirent of Sociology, 1971. 

McNulty, N. G. Training managers: The international guide. Scranton, 
Pennsylvania: Harper & Row, 1969, (ERIC ED 043 837.) 

McRuer, D. T. , Graham, F., & Krendel, E. S. Manual control of single 
loop systems: Part 1 and part 2. Journal of Franklin Institute , 
1967, 283 (1 & 2). 

Mee, J. F. Changing concepts of management. S. A. M . Advanced Management 
Journal, 1972, 37 (4), 22-35. 

Melton, A. Undergraduate pilot training in the USAF. M-32983-U, M588u, 
Professional study no. 3963, AUL/LRX, Maxwell AFB, AL 36112, 1970. 

Merriam, I. C. Concepts and measures of welfare. Proceedings of the 

Social Statistics Section of the American Statistical Association , 
Washington, D. C, 1967. 

Merwin, J. C. Evaluation designs and instruments. Paper Presented at the 
Symposium, The world of Evaluation Needs Re-Shaping, at the Annual 
Meeting of the American Educational Research Association, February 
1969. (ERIC ED 043 680.) 

Meyer, H. H. The validity of the in-basket test as a measure of managerial 
performance. Personnel Psychology , 1970, 23_ (3), 297-307. 

Meyer, H. H. & Walker, W. B. A study of factors relating to the effective- 
ness of a performance appraisal program. Personnel Psychology , 1961, 
14, 291-298. 

Michener, C. W. & Walzer, H. Developing community mental health volun- 
teer system. Social Work , 1970, 15 (4), 60-67. 



221 



35582-7-U, M6494m, Research study, AUL/LRX, Maxwell AFB, AL 36112, 
1972. 



ler, R. M. Close air support: Let's optimize the pilots. M-35582-7-U, 
M6493C, Research study no. 1335-71, AUL/LRX, Maxwell AFB, AL 36112. 

ler, S. M. & Bryce, H. J. The promotion of social mobility. Transactions 
of the Sixth World Congress of Sociology , 1966, 4^, 333-344. 

chell, J. M. & Mitchell, W. C. The changing politics of American life. 
In E. B. Sheldon & W. Moore (Eds.), Indicators of social change ; 
Concepts and measureraents , New York: Russell Sage Foundation, 1968. 

:off , I. I. & Betz, F. Dialectical decision theory: A meta-theory of 
decision-making. Management Science , 1972, 19 (1), 11-24. 

Ley, B. M. An analysis of student elimination in the military assistance 
conventional plot. M-35582-7-U, M6872a, Thesis no. 1725-67, AUL/LRX, 
Maxwell AFB, Al 36112, 1967. 

il cities training program. Chattanooga, Tennessee: Tennessee Univer- 
sity, 1969. (ERIC ED 043 847.) 

ie, D. J., Calhoon, R. , & O'Brien, J. K. Evaluation of a management 
development program. Personnel Psychology , 1964, j7, 431-440. 

s, A. The future oriented society: Axioms and methodology. Futures 
2, 1970, January, 312-316. 

225 















. .* 


'V. - • 






t 




0 









55 



h, D. H. & Inkeles, A. The CM scale: A comparative socio-psycholo- 
gical measure of individual modernity. Sociometry , 1966, 29, 353-377. 



h, R. G. Controlling the quality of training. Technical Report no. 
65-6, Contract with Department of the Army, Washington, D. C: George 
Washington University, Human Resources Research Office, 1965. 



42 



Moore, W. E. & Sheldon, E. B. Monitoring social change: A conceptual 
and programmatic statement. Proceedings of the Social Statistics 
Section , American Statistical Association , Washington, D. C, 1965. 

Morgan, J. N. & Smith, J. D. Measures of economic well-offness and their 
correlates. The American Economic Review , Proceedings and Papers, 
1970, 59, 450-462. 

Morsh, J. E. Identification of job types in the personnel career field. 
Technical Report no. PRL-TR-65-9, Project 7734, Task 773401, Lack- 
land AFB, Texas: Personnel Research Laboratory, Aerospace Medical 
Division, Air Force Systems Command, April 1965. 

Morsh, J. E. & Archer, W,. B. Procedural guide for conducting occupational 
surveys in the United States Air Force. Technical Report no. t'RL- 
TR-67-11, Lackland AFB, Texas: Personnel Research Laboratory Aero- 
space Medical Division, Air Force Systems Command, September 1967. 

Morsh, J. E. & Christal, R. E. Impact of the computer on job analysis in 
the United States Air Force. American Psychologist , 1966, 21 , 
676-677. 

Morton, J. A. Organizing for innovation : A systems approach to technical 
management . L. C. //T175.M66, New York: McGraw-Hill, 1971. 

Moser, C. A. Some general developments in social statistics. In M. Nissel 
(Ed.), Social trends . London: Her Majesty's Central Statistical 
Office, 1970. 

Moss, M. Consumption: A report on contemporary issues. In E. B. Sheldon 

& W. Moore (Eds.), Indicators of social change : Concepts and measure- 
ments . New York: Russell Sage Foundation, 1968. 

Motes, W. C. Rural development: Economic criteria for assessment of out- 
come and for research inputs. A Paper Presented at the Annual Meeting 
of the Rural Sociological Society, Denver, Colorado, August 1971. 

Mountcastle, R. C. The FB-lllA navigator: training a new co-pilot. M-35582- 
7-U, M928f, Research study, AUL/LRX, Maxwell AFB, AL 36112, 1972. 

Moyer, L. N., Kaplan, S. J., & Klein, I. A. Model cities: A design for 
evaluation. Social Science , 1970, 45 (2), 84-92. 



226 



43 



Mozzo, M. A. & Martinez, C. An approach to the determination of the edu- 
cational and training requirements for USAF logistics staff officers. 
SLSR-23-69, AUL/LRX, Maxwell AFB, AL 36112, 1971. 

Mukherji, P. N. a study in induced social change: An Indian experiment. 
Human Organization , 1970, 29^ (3), 169-177. 

Mullen, D. J. (EC.) Accountability. Athens, Georgia: Georgia University, 
Bureau of Educational Studies and Field Services, 1971. (ERIC ED 
057 507.) 

Murphy, J. R. Is it Skinner or nothing? Training and Development Journal , 
1972, 26 (2), 2-8. 

Murphy, K. R. Contrasts in spending by urban families. BLS Report no. 
238-8, Monthly Labor Review , 1964, 87_, 1249-1253. 

Murray, P. T. Tactical fighter training for the counterair mission. M- 
32983-U, M983t, Professional study, AUL/LRX, Maxwell AFB, AL 36112, 
1972 . 

Muse, W. R. Identificati on and development of tomorrow's managers. Per— 
sonnel Psychology , 1972, 51 (1)^ 46-91. 

Myers, J. T. ^^n analysis of zero defects. M-35582-7-U, M9963a, Thesis no. 
1825-67, AUL/LRX, Maxwell AFB, AL 36112, 1967. 

Nagel, D. M. The combat potential index, an improved combat effectiveness 
measure that allows attack aircraft to be included in ground battle 
war games. M-32983-U, N147c, Professional study, AUL/LRX, Maxwell 
AFB, AL 36112, 1972. 

NAHRO. Announcement of NAHRO management specialist training program. 
Journal of Housing; , 1971, 28.* 163. 

NAHRO. Announcement of NAHRO' s 1972 series of training and educational 
programs. Journal of Housing , 1971, 28, 594. 

NAHRO 's housing management specialist program moves into second year. 
Journal of Housing , 1972, 1, 27-30. 



227 



44 



National Communicable Disease Center of the Health Services and Mental 
Health Administration, Public Health Service, U. S. Department of 
Health, Education, and Welfare. Training methodology; Part I; 
Background theory and research ; An annotated bibliography . Wash- 
ington, D. C; U. S. GPO, 1969. 

National Wildlife Federation. 1971 EQ index. 1971 National Wildlife , 
Washington, D. C, 1971. 

Neale, J. G. Adult training; The use of programmed instruction. Occupa- 
tional Psychology , 1968, 42_ (4), 23. 

Nelson, C. C. An analysis of the workload imposed upon an instructor super- 
visor in the Air Training Command's technical training centers. M- 
35582-7-U, N425a, Research study no. 1430-71, AUL/LRX, Maxwell AFB, 
AL 36112. 

Nicholson, E. W. The performance of principals in the accountability 
syndrome. Paper Presented at National Association of Secondary 
Principals Annual Convention, March 1972. (ERIC ED 063 626.) 

The Non-Prof it Housing Center, Inc. Low and moderate income housing pro- 
duction and management. Vol. II. Washington, D. C, July 1972. 

Norman, W. T. Validation of personality tests as measures of trait-rating 
factors. Technical Documentary Report PRL-TDR-62-4, Project 7717, 
Task 771706, Lackland AFB, Texas: 6570th Personnel Research Labora- 
tory, Aerospace Medical Division, Air Force Systems Command, April 1962. 

North Carolina Department of Local Affairs. Training interaction. January 
1970. 

Northwestern University, Traffic Institute. Staff and command conference . 
A Course Packet, Chicago: Northwestern University, 1968. 

Northwestern University, Traffic Institute. Administration of police 

training . A Course Packet, Chicago: Northwestern University, 1971. 

Nugent, J. A. Methods to reduce cost and increase efficiency in UPT. M- 

35582-7-U, N967m, Research study, AUL/LRX, Maxwell AFB, AL 36112, 1972. 



228 



A5 



Occupational Survey Branch. United States Air Force job inventory: Heli- 
copter maintenance career ladder. AFPT-90-431-087 , Lackl.-^nd AFB, 
Texas, December 1, 1972. 

Occupational Survey Branch. United States Air Force job inventory: Safety 
specialist/technician career ladder. AFPT 90-241--099, Lackland AFB, 
Texas, December 1, 1972. 



Occupational Survey Branch. United States Air Force job inventory; Cook 

and diet therapy career ladders. AFPT 90-622-114, Lackland AFB, Texas, 
December 15, 1972. 



Occupational Survey Branch. United States Air Force job inventory: Defen- 
sive fire control systems career ladder. AFPT 90*»323-103, Lackland 
AFB, Texas, January 1, 1973. 



Occupational Survey Branch. United States Air Force job inventory: Main- 
tenance analysis specialist/ technician career ladder. AFPT 90-391- 
104, Lackland AFB, Texas, February 1, 1973. 

O'Connor, R. L. The future training role in the reserve structure. M- 
35582-7-U, 0184f, Research study, AUL/LRX, Maxwell AFB, AL 36112, 
1972. 



Odiorne, G. S. Training by objectives: An economic approach to manage- 
ment training. New York: The Macmillan Co., 1970. (ERIC ED 047 233.) 

Office of Economic Opportunity. Improvements needed in trainj^n^ and ::ech- 

nical assistance services provided to ant i-pover ty agencies . b-130515, 
Washington, D. C, April 1972. 

Office of the Governor, Division of Planning Coordination. Goals for Texas, 
phase one . Austin, Texas, September 1969 . 

Office of the Governor, Division of Planning Coordination- Goals for Texas, 
phase two . Austin, Texas , September 1970. 

Office of the Governor, Division of State-Local Relations* Community im- 
provement guide. Austin, Texas. (undated) 

Office of the Governor, Division of State-Local Relations* Texas communities 
tomorrow. Austirij Texas, October 1969. 



229 



46 



Office of the Governor, Division of State-Local Relations. Texas com- 
munities tomorrow. First Annual Report, Austin, Texas, November 
1970. 

Office of the Governor of Texas ^nd Texas Department of Community Affairs. 
Texas housing teport. Austin, Texas, January 1972. 

Office of Planning Coordination* Central New York regional economic study ; 
phase 1^: Basl_e inventorigg and analysis ^ Albany, New York, 1968. 

Ogburn, W. F. Social effects ol aviation . Cambridge, Massachusetts: 
Houghton Mifflin, 1946. 

Olson, M. Social Indicators and social accounts. Socio.-Economic Planning 
Science , 969, 2,, 335-346- 

Olson, M. Social indicators foi^ less developed countries. Draft, Ta/PM/M, 
Washington, D. C: Agenc/ for International Development, August 1971. 

Ontell, R. What is a social indicator? Unpublished manuscript, San Diego, 
California: Urban Observatory, 1971. 

Oregon State Civil Service Commission. Training in Oregon state government. 
A Report to tb^ Governor ^"d the 55th Legislative Assembly, Salem, 
Oregon, July 1968. 

Orientation and Training Branch, Division of Field Operations, Bureau of 

Hearings and Appeals. Letterwriting workshops, an evaluation study. 
Washington, D. C: Departnient of Health, Education, and. Welfare, 
April 1968. 

Osborn, W. C. & Goodman, B. E. A tentative organizational schema for deci- 
sion-making problems. Alexandria, Virginia: George Washington Univer- 
sity, Human Resources Research Office, July 1966. (ERIC ED 061 588.) 

Ostrom, E. On the meaning and measurement of output and efficiency and the 

production of urban police services. Unpublished manuscript, Blooming- 
ton, Indiana: Indiana University, Department of Politlc^jl Science, 1971. 

Ostrominski, J. Initiatives for deducing personnel and maintenance costs 

with job perfonnance aids. M-32983-U, 0851i, Professional study, AUL/ 
LRX, Maxwell aFB, AL 36112, 1972. 



230 



47 

Ostromlnski, J. job performance aids: Acceptance begins with under- 
standing. M-329B3~U, 085lj, Professional study, AUL/LRX, Maxwell 
AFB, AL 36112, 197,. 

Othling, W. L. Critique of war gaming, computer simulationi and game 
theory as tods in the study of military conflict situations. 
M-35582-7-U, 087c, Research study, AUL/LRX, Maxwell aFB, AL 36112, 
1972. 

Otto, C. P. & Glaser, R. o. The management of training : A handbook for 

training and j jevej-opment personnel . Reading, Massachusetts: Addison- 
Wesley Publishing Co., 1970. 

Overfelt, C. D. Professional military education and the USAF medical ser- 
vice. M-32983-U, 0962p, Professional study, AUL/LRX, Maxwell AFB, 
AL 36112, 1972. 

Palley, H. A. & Palley, M. L. Social policy analysis the use of social 

indicators. Wgl£_ar^ in Review , 1971, _9, 8-14. 

Papaloizos, A. & Stiefel, R. The effectiveness of management teaching 
methods. ManjLg^ement Decision , 1971, _9 (Summer), 111-121. 

Parkhurst, N. P. a study to examine the need for improving air force pilot 
utilization policies, develop alternatives that, enhance pilot pro- 
curement and retention while minimizing training costs. M-35582-7-U , 
P246s, Research study, AUL/LRX, Maxwell AFB, AL 36112, 1972. 

Parr, R. S. Improved effectiveness for air combat maneuvering training: 
A case study. M-32983-S, P259Li, Professional study no. 3471, AUL/ 
LRX, Maxwell APR, AL 36112, 1967. 

Passett, B. A. Leadership development for public service. Houston, Texas: 
Gulf Publishing Co., 1971. (ERIC ED 056 265.) 

Peach, L. Personnel management — art or science? Personnel journal, 1972, 
A (2), 26-29. 

Pearl, A. New careers and model cities. Bethesda, Maryland: ERIC Document 
Reproduction Service, January 1968. (ERIC ED 039 453j 

Pena, W. M. & Focke, vJ. W. Problem seeking. New directions in architectural 
programming. Houston, Texas: Caudell, Rowlett & Scott, 1969. 
(ERIC ED 037 9^0.) 



231 



48 



Perle, E. D. Notes on urban information systems and urban indicators. A 

Paper Presented at the Annual Meeting of the Association for Computing 
Machinery, New York, September 1970. 

Perle, E. D. Local societal indicators. A Progress Report. Proceedings 
of the Social Statistics Section , American Statistical Association , 
Washington, D. C, 1971. 

Peti-rsen, P. B. Leadership training. Training and Development Journal , 
1972, April, 38-42. 

Pfaff, A. B. An index of consumer satisfaction. A Paper Presented at the 
Annual Meeting of the American Economic Association Jointly with the 
Association for the Study of the Grants Economy, New Orleans, Louisi- 
ana, December 1971. 

Piele, P. K. New programs for training school administrators. Analysis of 
literature and selected bibliography. Analysis and Bibliography 
Series no. 10, Eugene, Oregon: Oregon University, ERIC Clearinghouse 
on Educational Administration, September 1970. (ERIC ED 043 119.) 

Pitts, G. & Holmes, R. T. The total cost approach to cost effectiveness. 
SLSR-3-69, AFIT/L, Wright-Patterson AFB, OH 45433, 1970. 

Pittsburgh University, University Urban Interface Program. The impact of 
the University of Pittsburgh on local economy. DHEW //BR-8-0725, 
Washington, D. C, April 1972. (ERIC ED 063 901.) 

Place, H. C. Undergraduate pilot training: A new approach to an old pro- 
gram. M--35582-7-U, P697a, Research study, AUL/LRX, Maxwell AFB, AL 
36112, 1972. 

Popham, W. J. (Ed.) Criterion-referenced measurement : An introduction . 

Englewood Cliffs, New Jersey: Educational Technology Publications, 
1971. 

Porter, K. W. Managerial attitudes and performance . Homewood, Illinois: 
Irwin, 1968. 

Potter, R. G., Jain, A. K., & McCann, B. Net delay of next conception by 
contraception: A highly simplified case. Population Studies , 1970, 
24 (2), 173-192. 



232 



49 



Powell, E. Do your own thing. M-82983-U, P8821cl, Trof <»ms lon.i I Mlu.ly no. 
3983, AUL/LRX, Maxwell AFB, AL 30112, 1970. 

Powers, J. G. The air force pilot's exodus to the airlines still a 

problem in the 1970's? M-35582-7-U, P8882a, Thesis no. 2045-67, 
AUL/LRX, Maxwell APB, AL 36112, 1967. 

Prather, R. L. Job profile, perfomance evaluation, and career progress. 
Personnel Journal , 1969, 48 (7), 513-522. 

Prien, E. P. & Kult, M. Analysis of performance criteria and comparison 

of a priori and empirically derived keys for a forced choice scoring. 
Personnel Psychology , 1968, 2j., 505-513. 

Prien, E. P., Barrett, G. , & Sretlik, B. Use of questionnaires in job 
evaluation. Journal of Industrial Psychology ^ 1965, 2i 91-94. 

Pucel, D. J. et al. The ability of standardized test instruments to pre- 
dict training success and employment success. Project Mini-Score, 
Final Technical Report, Minneapolis: Minnesota University, March 1972. 

Quimby, R. F. Can all air force officers really be counselors? M-35582-7- 
U, Q63c, Research study, AUL/LRX, Maxwell AFB, AL 36112, 1972. 

Quisenberry, J. D. An analysis of ANG technical training. M-35582-7-U, 

Q8a, Research study no. 1110-70, AUL/LRX, Maxwell AFB, AL 36112, 1970. 

Rapp, M. The analytical aspects of evaluating on-goin^, programs. Paper 

Presented at American Educational Research Association Annual Meeting, 
Chicago, 1972. 

Rappaport, C. .A framework for evaluating the impact of the model cities 

program. A Paper Presented at the National Meeting of the Operations 
Research Society of America, Detroit, October 1966. 

Ratajczak, R. Problems in the evaluation of poverty programs. Santa Monica, 
California: Systems Development Corporation, 1969. 

Raymond, R. S. & Richards, E. Social indicators and business u^icisions. 
Michigan State University Business Topics , 1971, Autumn, 42-46. 

Record, J. F. An analysis: The training role of the United States Air Force 
instru^^ent pilot instructor school. M-35582-7-U, R311a, Research 
study, AUL/LRX, Maxwell AFB, AL 36112, 1972. 



233 



50 



Relcliart, R. D. Fighter aircrew preparailon for Souihenst Asia. M- 
3:;582*7-U, R348f, Thesis no. 1015-68, AUL/LRX, Maxwell ATH, AL 
36112, 1968. 

Repput, J. W. C--141 combat airlift navigator training in the military 
airlift conunand. M-35582-7-U, R425c, Research study, AUL/LRX, 
Max\ Jll AFB, AL 36112, 1972. 

Rhinehart, J. B. et al. Comparative study of need satisfactions in govern- 
mental and business hierarchies. Journal of Applied Psychology , 
1969, 53, 3 (part 1), 230-235. 

Riccobono, J. A. & Cunningham, J. W. Work dimensions derived through 
systematic job analysis: A study of the occupation analysis in- 
ventory. North Carolina State University, National Center for 
Educational Research and Development, 1971. (ERIC ED 062 541.) 

Rice, E. T. An exploratory study to investigate the aspects of the con- 
struction and use o. instruments encompassing the evaluative strate- 
gies of context, inout, process, and product of in-service programs. 
Iowa City: Iowa University, Center for Research in School Administra- 
tion, April 1969. (ERIC ED 043 681.) 

Richard, R. Subjective social indicators. Chicago: National Opinion 
Research Center, University of Chicago, 1969. 

Ridley, C. E. & Herbert, S. Measuring municipal activities: A survey 

of suggested criteria and reporting forms for appraising administra- 
tion. Chicago: The International City Managers* Association, 1938. 

Rindahl, G. F. Using programmed learning to improve power plant reliability. 
M-35582-7-U, R579u, Thesis no. 2120-67, 1967. 

Rivlin, A. M. Systematic thinking for social action . Washington, D. C; 
Brookings Institution, 1971. (ERIC ED 057 140.) 

Roberts, E. E. Increased leadership effectiveness. M-32983-U, R643i, 
Research Report no. 3672, AUL/LRX, Maxwell AFB, AL 36112, 1968. 

Rokeach, M. & Parker, S. Values as social indi^itors of poverty and race 

relations in America. The Annals of the American Academy of Political 
and Social Science , 1970, 388 , 97--111. 



23 i 



51 



Ronan, W. W. A factor analysis of II job performance measures. Per- 
sonnel Psychology , 1963, 1^, 255-267. 

Ronan, W. W. Evaluation of skilled trades performance predictors. Educa- 
tional and Psychological Measurement , 1964, 24_, 601-608. 

Ronan, W. W. Evaluation of three criterial management performance. Journal 
of Industrial Psychology , 1970, _5 (1), 19-29. 

Rosenthal, R. A. & Weiss, R. S. Problems of organizational feedback processe 
In R. A. Bauer (Ed.), Social indicators . Cambridge, Massachusetts: 
Institute of Technology Press, 1969. 

Rossi, P. H. Coinraunity social indicators. DHEW, //BR-6-1610, Baltimore, 
Maryland: Johns Hopkins University, Center for the Study of Social 
Organization of Schools, October 1970. (ERIC ED 043 090.) 

Ruckeishaus, W. D. From awareness to action. Paper Presented at the Second 
Earth Day Celebration, Ohio State University, Columbus, April 22, 1971. 
(ERIC ED 052 980.) 

Russett, B. M. Indicators for America's linkages with the changing world. 
The Annals of the American Academy of Political and Social Science , 
1970, 288, 82-96. 

Russett, B. M. & Meyer, R. B. World handbook of political and social indi- 
cators . New Haven: Yale University Press, 1964. 

Safka, H. I. The city as a social system: Implications for planning and 

urban policy. Editor's introduction. American Behavioral Scientist , 
1972, 15 (4), 483^493. 

Sagadencky, E. Staff management. In Planning 1966 * Chicago: American 
Society of Planning Officials, 1966. 

Sametz, A. W. Production of goods and services: The measurement of econ- 
omic growth. In E. B. Sheldon & W. E. Moore (Eds.), Indicators of 
social change : Concepts and measurements. New York: Russell Sage 
Foundation, 1968. 

Sanders, J. H. A special training project for evaluators. Final report, 
Lincoln Public Schools, Nebraska, August 1969. (ERIC ED 047 422.) 



23j 



52 



Sandliu, H, S. & Allen, D. \i. The prison guard: Job perceptions and in- 
service training in India. Indian Journal of Social Work , 1971, 
32 (2), 115-120. 

Sandler, D. T. An analysis of educational technology and pre-commissioning 
training at officer training school, USAF. M-35582-7-U, S2171a, 
Thesis no. 1050-68, AUL/LRX, Maxwell AFB, AL 36112, 1968. 

Sardy, S. & Sardy, H. Administrating decision-making and resource alloca- 
tion. Paper Presented at the American Educational Research Associa- 
tion Annual Meeting, New York, February 1971. (ERIC ED 048 629.) 

Sawchuck, R. & Gitter, A. G. Eight subjective indicators of quality of 
life. Report no. 51, Boston: Boston University, Communication 
Research Centex*, 1971. 

Sax, R. M. The planning process for social and physical and the need 
for an environmental design. Monograph no, 2, Pleasant Hill, 
California: Contra Ccsta County Department of Education, November 
1965. (ERIC ED 028 614.) 

Scammon, R. M. Electoral participation. The Annals of the American 
Academy of Political and Social Science , 1967, 371 , 59-71. 

Scarboro, G. L. An analysis of the drug education programs in the USAF 
pre-commissioning and professional military educational schools. 
M-35582-7-U, S2851a, Research study no. 1695-71, AUL/LRX, Maxwell 
AFB, AL 36112. 

Schick, A. From analysis to evaluation. Annals of the American Academy 
of Political and Social Science , 1971, 394 , 57-71. 

Schindler-Rainman, E. & Lippitt, R. The volunteer community: Creative 
use of human resources. Washington, D. C«: National Training 
Labs, Institute for Applied Behavioral Science, 1971. (ERIC 053 
038.) 

Schneeberger , K. C. Gaming as an instrument of farm management educa- 
tion — a development and evaluation. Stillwater, Oklahoma: Oklahoma 
State University, 1968. (ERIC ED 036 685.) 

Schneier, E. The intelligence of Congress: Information and public policy 
patterns. The Annals of the American Academy of Political and Social 
Sci3nce, 1970, 388, 14-29. 

o 230 
ERIC 



53 



Schools Council. Coniraunity service and the curriculum. London, 
England, 1968. (ERIC ED 049 135.) 

Schultz, D. G. The anlaysis of job performance by multidimen? jii. 
scaling technique. Journal of Applied Psychology , 1964 
329-335. 

Schultz, D. G. & Siegel, H. I. The analysis of job performance by multi- 
dimensional scaling techniques. Journal of Applied Psychology , 
1964, 329-335. 

Schultz, D. G., Fische, M. A., Lanterman, R. S., & Siegal, A. I. Absolute 
scaling of job performance. Journal of Applied Psychology , 1968, 
5^, 313-318. 

Schwartz, A. PPBS and evaluation research: Problems and promises. In 
E. E. Schwartz (Ed.), Planning-^rogramming-budgeting systems and 
social welfare . Chicago: The University of Chicago, The School of 
Social Service Administration, 1970. 

Schwartz, M. A. Trends in white attitudes toward Negroes. Chicago: The 
University of Chicago, National Opinion Research Center, 1967. 

Scott, L. C. & Blume, P. R. Some evidence on the economic effectiveness 
and institutional versus on-the-job training. Social Science 
Quarterly , 1971, 51 (4), 910-923. 

Sears Roebuck and Company Personnel Office, National Sears Training, Retail 
Induction Training. Turn and learn for successful selling . No. 11588 
(F2952 Rev. 1/70), Department 707 Parent, Austin, Texas. (undated) 

Sedlak, G. The management value of primetime. M-35582-7-U, S449m, 
Thesis no. 2290-67, AUL/LRX, Maxwell AFB, AL 36112, 1967. 

Shapero, A. The official organization watchers guiue. In T. G. Harris 

(Ed.), Careers today . Del Mar, California: Communications/Research/ 
Machines, Inc., January 1969. (charter issue) 

Shariff, Z. Social information and government-sponsored development: A 

case study from West Pakistan. The Annals of the Academy of Political 
and Social Science , 1971, 393 , 92-108. 



EKLC 



237 



Sheffieck, C, F. Management training and evaluation by major United 

States industrial corporations. Detroit: Wayne Stac;^ University, 
1969. (ERIC ED 041 243.) 

Shelburne, J. C. & Graves, K. J. Education in the armed forces > New 
York: The Center for Applied Research, Inc., 1965. 

Sheldon, E. B. & Freeman, H. E. Notes on social indicators: Promises 
and potential. Policy Sciences , 1970, _1, 97-111. 

Sheldon, E. B. & Moore, W. E. (Eds.) Indicators of social change . New 
York: Russell Sage Foundation, 1968. 

Shinn, A. M. Magnitude estimacion: Some applications to social indica- 
tors. A Paper Presented at the Annual Meeting of the American 
Political Science Association, Chicago, Illinois, September 1971. ( 

Shultz, J. & Shultz, M. An annotated review of the literature on 

volunteering. Washington, D. C: Center fur a Voluntary Society, 
National Institute for Applied Behavioral Science, 1970. 

Siegel, A. I., Schultz, L. G., Fischl, M. A. , & Lanterman, R. S. 

Absolute scaling of job performance. Journal of Applied Psychology 
1968, 52, 313-318. 

Silberman, H. F. Design objectives of the instructional management 

system. A Professional Paper, Santa Monica, California: System 
Development Corporation, February 1968. 

Simon, R. J. Public attitudes toward population and pollution. The 
Public Opinion Quarterly , 1971, 35^y 93-99. 

Skovgaard, R. A. A ready force of forward air controllers. M-35582-7-U, 
S6284r, Research study, AUL/LRX, Maxwell AFB, AL 36112, 1972. 

Sloan, H. W. Social change, quality of life, and urban affairs: A 

selective bibliography of current popular literature. Nashville, 
Tennessee: Center for Community Studies, 1970. 

Small Business Administration. Administrative management course program: 
Communication and control . Topic 9. Washington, D. C, 1965. 

2 3 S 



55 



Smith, D. H. & Inkeles, A. The OM scale: A comparative socio-psycholo- 

gical measure of individual modernity. Sociometry , 1966, 29^, 353-377. 

Smith, R. G. Controlling the quality of training. Technical Report no. 

65-6, Contract with Department of the Army, Washington, D. C: George 
Washington University, Human Resources Research Office, 1965. 

Smith, R. G. Design of instructional systems. Technical Report no. 66-18, 
Department of the Army Contract no. DA 44-188-AR0-2, Project no. 
2J024701A712 01, Washington, D. C: George Washington University, 
Human Resources Research Office, 1966. 

Smith, R. G. The engineering of educational and training systems . Lex-- 
ington, Massachusetts: D. C. Heath & Co., 1971. 

Smith, R. L. Evaluation of air national guard field training sites. 

M-32983-U, S6583e, Professional study no. 4027, AUL/LRX, Maxwell 
AFB, AL 36112, 1970. 

Smode, A. F. Hi nan factor inputs to the training device design process. 
Report no. NAVTRADEVCEN-69-C-0298-1, Orlando, Florida: Naval 
Training Device Center, September 1971. (ERIC ED 057 606.) 

Snyder, A. E. Provide squadron commanders centralized guidance and special- 
ized training to cope with contemporary people problems. M-35582- 
7-U, S6754p, Research study, AUL/LRX, Maxwell AFB, AL 36112, 1972. 

Social Science Research Council. Social indicators newsletter. Washington 
D. C: SSRC Center for Social Indicators, March 1973. 

Soderlind, S. E. The outlook: Appraisal of current trends in business 
and finance. Wall Street Journal , 1970, 51^ (14), 1. 

Sorem, D. N. The use of military helicopters in civilian medical evacua- 
tion roles. M-35582-7-U, S7132u, Research study, AUL/LRX, Maxwell 
AFB, AL 36112, 1972. 

Speroff, B. J. Experimental procedures and techniques for evaluating 

supervisory performance. Personnel Administration, 1961, _24 (3), 
4-10. 

Spivey, R. C. Realistic tactical training. M-35582-7-U, S7611r, Research 
study, AUL/LRX, Maxwell AFB, AL 36112, 1972. 



239 



56 



Springer, M. (Ed.) Political intelligence for America's future. The 
Annals of the American Academy of Political and Social Science , 
1970, March, 388 . 

Springer, M. Social indicators, reports, and accounts: Toward the man- 
agement of society. The Annals of the American Academy of Political 
and Social Science , 1970, 388 , 1-13. 

Stadler, J. T. Training the UPT in tactical fighters. M-35582-7-U, 
S7771t, Research study, AUL/LRX, Maxwell AFB, AL 36112, 1972. 

Stanford Research Institute. Toward master social indicators. SRI 

Project 6747, Research Memorandum EPRC-6747-2, Menlo Park, Califor- 
nia: SRI Educational Policy Research Center, 1969. 

Stewart, W. J. Determining first-line supervisory training needs. Training 
and Development Journal , 1970, 24 (4), 12-19. 

Stine, V. L. An analysis of agri-business management functions for personnel 
training. East Lansing, Michigan: Michigan State University, College 
of Agriculture and Human Resources, 1969. (ERIC ED 045 921.) 

Stockdale, J. D. & Aronson, S, Technology and societal change: Selected 
sources with emphasis on resource allocation. Ithaca, New York: 
Cornell University, 1970. 

Story, W. H. & Waltman, R. P. A study of the education of logistics 

officers with regard to the use of computer simulation in logistics 
planning. SLSR-49-67, AFIT/L, Wright-Patterson AFB, OH 45433, 1968. 

Stratton, R. P. & Brown, R. Improving productive thinking by training in 
production and judgement. Paper Presented at the Annual Meeting 
of the American Educational Research Association, New York, February 
1971. (ERIC ED 046 995.) 

Strauss, G. Management by objectives: A critical view. Training and 
Development Journal , 1972, April, 10-15. 

Street, P. Community action in an Appalachian county, slower but surer? 
Lexington, Kentucky: University of Kentucky, 1968. 



210 



57 



Studer, R. G. Human systems design and the management of change. A 
Paper Presented at the Second International Conference on the 
Problems of Modernization in Asia and the Pacific, Honolulu, 
Hawaii, East-West Center, August 1970. 

Stufflebeam, D. L. revaluation as enlightenment for decision-making. 

Columbus, Ohio: Ohio State University, Evaluation Center, January 

1968. (ERIC ED 048 333.) 

Stufflebeam, D. L. Design of a planning and assessment system for the 

division of manpower and institutions. Washington, D. C: National 
Center for Educational Research and Development, June 1971. (ERIC 
ED 062 386.) 

Sule, 0. Interaction betwsen technological and social changes. Futures , 

1969, ]., 402-407. 

Tagliere, D. A. Organalysis. Training and Development Journal , 1972, 
April, 30-36. 

Taylor, J. F. Planning and organization for the C-5A aircrew training. 

M-35582-7-U, T2432-7-U, T2432p, Thesis no. 2520-67, AUL/LRX, Maxwell 
AFB, AL 36112, 1967. 

Taylor, W. L. A continuing program of chaplain education. 358.34, A2988s, 
Student report no. 26, AUL/LRX, Maxwell AFB, AL 36112, 1968. 

Teague, R. L. Communicy power and social change: A case for social action 
with implications for occupational education. Raleigh, North Carolina 
North Carolina State University, Center for Occupational Education, 
1969. (ERIC ED 055 223.) 

Terleckyj , N. E. Data systems for measuring social change. A Paper 

Presented at the Annual Meeting of the American Statistical Associa- 
tion, Detroit, Michigan, 1970. 

Terleckyj, N. E. Measuring possibilities of social change. Looking Ahead . 
1970 (18), 1-11. 

Texas A and M University, Engineering Extension Service and Texas Depart- 
ment of Community Affairs. Evaluation study of principles of super- 
vision. College Station, Texas: Texas A and M Uni /ersity, September 
1972. 



211 



58 



Texas A and M University, Engineering Extension Service, and Texas 
Education Agency, Industrial Education Service. Suggested 
basic course outline for police administration. College Station: 
Texas A and M University. (undated) 

Texas Department of Community Affairs, Center for Social Policy and 

Community Development. Model curricula and training techniques for 
use in training sub-professional employees of public housing author- 
ities. Philadelphia: Temple University, School of Social Admin- 
istration, June 1971. 

Texas Department of Community Affairs, Community Services Division. 

Education and training information bulletin. Austin, Texas, 1972. 

Texas Department of Community Affairs, Division of Comprehensive Planning. 
A methodology report on the small communities development program. 
Austin, Texas, January 1972. 

Texas Department of Community Affairs. Management assistance study: 
Texas model cities program. Austin, Texas, March 1972. 

Texas Department of Public Safety. Management workshop [upper echelons 
of management]. Given August 30-September 3, 1971 and September 
27-October 1, 1971, Austin, Texas. (undated) 

Texas Department of Public Safety, Programmed Instructional Units. The 
history of police : Programs and_ //2 . Austin, Texas. (undated) 

Texas Department of Public Safety, Programmed Multi-Media Instructional 
Units. Police-community relations : Programs //I, //2, //3, //4. 
Austin, Texas. (undated) 

Texas Department of Public Safety. Training material hand-outs for 

first-line supervisors. A Collection of Material, Austin, Texas. 

Texaf> Education Agency, Adult and Continuing Education Division. Evaluatio 
institutional training MDTA. An Evaluation Instrument, Austin, 
Texas . (undated) 



Texas Education Agency. Program visit report adult and continuing educa- 
tion. Austin, Texas. (undated) 



.S9 



Texas Municipal League. Survey of training needs and evaluation of the 
1970-71 Title VIII training program. Austin, Texas: Texas 
Municipal League for the Texas State Department of Community Affairs 
(undated) 

Texas State Technical Institute. Evaluation report of community develop- 
ment training programs: Technical manpower and the disadvantaged. 
Report no. CD 70-517, Waco, Texas, 1971. 

Texas Urban Development Commission. Urban Texas: Policies for the future 
Austin, Texas, November 1971. 

Texas Water Quality Board. Report on training grant no. 638.1.01 author- 
ized under section 5(g) (1) federal water pollution control act. 
Submitted to Environmental Protection Agency, Dallas, Texas, 
August 31, 1972. 

Thomas, P. J. An evaluation of methods for predicting job performance of 
personnelmen. Report no. STB-72-4, San Diego: Naval Personnel and 
Training Research Lab, September 1971. (ERIC ED 058 524.) 

Thorley, S. Evaluating an in-company management training program. Train- 
ing and Development Journal , 1969, 23 (9), 48-50. 

Thorley, S. What can we do about evaluating management development? 
Training in Business and Industry , 1972, £ (2), 34-36. 

Timpane, P. M. Hard lessons in the assessment of social action programs: 
The case of ESEA Title I. Washington, D. C.: Deaprtment of Health, 
Education and Welfare, 1972. 

Tinney, R. E. et al. Education in a new era: Innovations today for 

security tomorrow. M-32983-U, T591e, Professional study, AUL/LRX, 
Maxwell AFB, AL 36112, 1972. 

r Lesley, J. S. The Apollo 204 accident: A search for management lessons. 
M--35582-7-U, T5921a, Thesis no. 1250-68, AUL/LRX, Maxwell AFB, AL 
36112, 1968. 

Toffler, A. The art of measuring the arts. The Annals of the Ajnerican 
Academy of Political and Social Science, 1967, 373, l4l-l5S. 



60 



Tracor, Inc. A Comprehensive evaluation of the community action program 
in Austin and Ttavis County> Texas. Final Report, Austin, Texas, 
1969. (ERIC ED 040 315.) 

Traf £3* . Institute. Wli course. Evanston, Illinois: Northweo eru 
Jni-Versity . 

Training, Research and Developments* Inc. Career ladder and curriculum 
guide for code enforcement i'^spector. Washington, D. C. 

Training, Research and Developmental Inc. Career ladder and curriculum 
guide for zoning inspector. Washington, D. C. 

Training, Research and Developments* Inc. Handbook for training code 
enforcement and zoning inspection personnel. Washington, D. C. 

Trocki, K. F. et al. An explorat^^^^y assessment of CAP evaluation 

activities. Volume 2. Catalog of Abstracts of Evaluation Studies. 
Pittsburgh: American Institutes for Research, 1969, (ERIC ED 
038 609.) 

Truett, H. W. Systems management ^or air force training. M-32983-U, 
T86645, Professional study no. 3528, AUL/LRX, Maxwell AFB, AL 
36112, 1967. 

Tugac, N. Social indicators and social prediction in planning process: 
A review of literature and bibliography. State Planning Organiza- 
tion Publishing no* DPt: ll24 — SPD: 243, Ankara, Turkey: Devlet 
Planlama Teskilati, BakanliKlar, 1971. 

Turner, A. F. The army ROTC — yesC^tday^ today and tomorrow. M-32983-U, 
T944la, Research Report no. 3697, AUL/LRX, Maxwell AFB, AL 36112, 
1968. 

Turney> J. R. Expectancy ^ valence and intrinsic activity value as predictors 
of motivation in a technical'^prof essional organization. Proceedings 
£f_ the^ Annual Convention of^ the American Psychological Association ^ 
1972, I (1), 453. 

Ulrich, G. Indicators of equal opportunity for social mobility in the 

occupational structure. Unpublished manuscript. Ann Arbor, Michigan: 
University of Michigan, School of Social Work, 1969. 



EKLC 



214 



61 



United Nations. Comp^ndiuni of social statistics: 1963. New York, 1963. 

United Nations. Compendium of social statistics: 1967. New York, 1968. 

United Nations Research Institute for Social Development. The concept of 
development and its measurement. International Social Deve lopment 
Review, 1970, 2^, 1-6. 

United Nations, Social Development Division. Social policy and the dis- 
tribution of Income in the nation. Ekistics , 1969, 28^, 399-^05. 

U. S. Army Southeast Signal School. Training evaluation plans > Ft. 
Gordon, Georgia, 1969. 

U. S. Civil Service Commission: Bureau of Training. Employee trai ning 
in the federal getyjce. FY1968, Washington, D. C, February 1969, 
CS 1.48: T-7. 

U. S. Civil Service CoMiission: Bureau of Training. Application of a 
systems approach to training. A case study training systems and 
technology series: No. II. Washington, D. C.: U. S. Government 
Printing Office, June 1969. (ERIC ED 058 517.) 

U. S. Civil Service CoH^^ission: Bureau of Training. Instructional sys- 
tems and technology: An introduction to the field and its use in 
federal training. Training Systems^ and Technology , Series no. I, 
June 1969. 

U. S. Civil Service Coi^niission: Bureau of Training. Studies and reports 
relating to trailing and education, fiscal year, 1969. CS 1,48: 
T-6/2. 

U. S. Civil Service Coiuniisslon: Bureau of Training. Studies and reports 
relating to trailing and education. T-6, April 1970. 

U. S. Civil Service Con^iission. Job grading system for trades and labor 
occupations . CS 1.41/4: 512-11, trans. 12, Washington D, C, 
June 1970. 

U. S. Civil Service CoiiUiiission. Interagency training programs. Dallas 
Region, 1971-1972. 



62 



S. Civil Service Conimission, Management and )^roup performance , a^ 
ji^uide f or^ the use of instructors who conduct training for managers 
at the operating level . Vol, I (revised), CS17/4: M31/2/1971, 
Washington, D. C., 1971. 

l^- S. Civil Service Commission. Management and group performance , a guide 
for the_ use of instructors who conduct training for managers at the 
op erating level . Vol. II (revised), CSl/74: M31/2/1971, Washing- 
ton, D. C. , 1971. 

IJ- S. Department of Health, Education, and Welfare. The licensed 

practical nurse as charge nurse . Instructor's Guide and Trainee's 
Guide, Rockville, Maryland: Public Health Service, May 1972. 

S. Department of Housing and Urban Development. Urban and regional 
information systems: Support for planning in metropolitan areas. 
Washington, D. C.: U. S. Government Printing Office, 1968. 

S. Department of Housing and Urban Development. Urban intern train- 
ing^ . A HUD handboo k. Washington, D. C. , November 1970. 

IJ- S. Department of Housing and Urban Development. Files management , 

general instructions . A HUD handbook . Washington, D. C, April 1971. 

IJ- S. Department of Housing and Urban Development. Management assistance 
programs and information systems , Washington, D. C., January 1972, 
HH 1.42: 8. 

IJ- S. Department of Housing and Urban Development. A training model for 
low rent housing management . Washington, D. C, 1972. 

IJ- S. Department of Labor, Bureau of Labor Statistics. Three budgets 

for a retired couple in urban areas of the United States, 1967-68. 
Bulletin, 1970-6, Washington, D. C: U. S. Government Printing 
Office, 1970. 

U* S. Department of Labor. USTES employer services training resource. 

Master Guide and ESTR Series 100, 200, 3o0, 40C, and 900 Guides and 
Trainee Materials, Contract no. 34-0-0400-000, Washington, D. C: 
Manpower Administration, United States Training and Employment Ser- 
vice, April 1971. 



216 



63 



University of Louisvill Southern Police Institute. Administrative 
officers course. A brochure, Louisville, Kentucky, 1972. 

University of Louisville, Southern Police Institute. School of police 
admini .ration. A brochure. Louisville, Kentucky, 1972. 

University of North Carolina. Local government administrator's course. 

Vol. II, Chapel Hill, North Carolina: Institute of Government, 1970. 

University of North Carolina. Local government administrator's course. 

Vol. Ill, Chapel Hill, North Carolina, Institute of Government, 1970. 

Valen, R. D. Teacher training for faculty duty at squadron officer school. 
M-35582-7-U, V152t, Research study no. 1328-70, AUL/LRX, Maxwell 
AFB, AL 36112, 1971. 

Vanderwaal, G. W. Undergraduate pilot training: A new method. M-35582- 
7-U, V242u, Research study, AUL/LRX, Maxwell AFB, AL 36112, 1972. 

Vanecko, J. L. & Jacobs, B. The impact of the community action program 
on institutional change: assistance to community organization as 
a successful strategy. Final Report, Chicago: National Opinion 
Research Center, May 1970. (ERIC ED 044 582.) 

Venable, J. C. & Romer, R. A. An analysis of the impact of tactical re- 
organization on avionics maintenance training. SLSR-15-67, AFIT/L, 
Wright-Patterson AFB, OH 45433^ 1968. 

Verba, S. Democratic participation. The Annals of the American Academy 
of Political and Social Science . 1967, 373 . 

Verborgh, R. Teaching mininf mastery and mine security: Some methodolo- 
gical reflections. Revue de I'instituc de Sociologie (Solvay), 1971, 
1, 59-66. 

Veri, C. C. & Vender Haar, T. A. Training the trainer. St. Louis: Missouri 
University, Extension Division. (undated) (ERIC ED 048 574.) 

Vestermark, S. D. Social indicators of social effects and the social in- 
ventory after attack. Symposium Presented at Meeting of Advisory 
Committee on Civil Defense and Office of Emergency Planning, Fort 
Monroe, Virginia, November 1965. 



217 



64 



Vine, N. R. Visual illusions — a cause of aircraft accidents. M- 

35582-7-U, V782v, Research study, AUL/LRX, Maxwell AFB, AL 
36112, 1972. 

Viteles, M. S. Program of humanistic studies for Bell System executives, 

University of Pennsylvania (1953-60), long-range impact on manager- 
ial attitudes and behavior: A follow-up survey. Pennsylvania: 
Bell of Pennsylvania, 1969. 

Vonder Haar, T. A« Performance standards for managers. Personnel 
Journal , 1972, 51 (1), 27-31. 

Vroom, V. H. Ego involvement, job satisfaction and job performance. 
Personnel Psychology , 1962, 15, 159-177. 

Waad, C. E. Error analysis simplified. M-32983-U, W874e, Professional 
study, AUL/LRX, Maxwell AFB, AL 36112, 1972. 

Waddell, R. Selection and utilization of test pilots in the United 

States Army. M-35582-7-U, W176s, Research study no. 1345-70, AUL/ 
LRX, Maxwell AFR, AL '.f.lJ2, 1970. 

Walker, A. E. A ^easu:-'-ment stern for future undergraduate pilot train- 
ing. M-3538 .i77im. Research study, AUL/LRX, Maxwell AFB, 
AL 36112, 197^. 

Wallace, R. C. & Shave2 a, R. ..\ A systems analytic approach to eval- 
uation: A ."leuristic odel anc its application. Syracuse, New 
York: Eastern Regional Institute for Education, 1970. (ERIC ED 
058 292.) 

Walter, C. W. A survey and discussion of education required to prepare 

new air force officers for effective assumption of supervisory duties 
as required under the resources management systems (R>IS) concepts. 
SLSR-14-71B, AFIT/L, Wright-Patterson AFB, OH 45433, 1972. 

Walters, M. L. Pilots at a lower cost. M-32983-U, W2354p, Professional 
study no. 4255, AUL/LRX, Maxwell AFB, AL 36112. 

Wardrop, J. L. Determinin3 most probable causes: A call for re-examining 

evaluation methodol gy. New York: American Educational Research Asso 
ciation, February 1971. (ERIC ED 048 337.) 



213 



65 



Wasson, K. W. & O'Connell, H. J. Social indicators: The bibliographic 
review as a first step toward scientific inquiry. Unpublished 
paper, Ames, Iowa: Iowa State University, Department of Sociology, 
•971. 

Webb, W. B. An analysis of junior officer manager development in the 
SAC operational wing. M-35582-7-U, W368a, Thesis no. 2770-67, 
AUL/LRX, Maxwell AFB, AL 36112, 1967. 

Weinman, B., Sanders, R. , Kleiner, R. , & Wilson, S. Commu ity based 

treatment of the chronic psychotic. Community Mental Health Journal 
1970, 6 (1), 13-21. 

Welsh, D. A. Computer-assisted instruction in military and industry. 

M-35582-7-U, W461c, Thesis no. 1310-68, AUL/LRX, Maxwell AFB, AL 
36112, 1968. 

Wenig, R. E. & Wolansky^ W. D. Review and synthesis of literature on job 
training in industry. Columbus, Ohio: Ohio State University, The 
Center for Vocational and Technical Education, June 1972. 

Wenstrand, A. P. A study of air combat maneuvering training in the U. S. 
Air Force. M-35582-7-U, W476s, Research study no. 2100-71, 
AUL/LRX, Maxwell AFB, AL 36112. 

West, C. B. An analysis of junior rated officer retention in tactical 

air command. M-35582-7-U, W516a, Thesis no. 2790-67, AUL/LRX, Max- 
well AFB, AL 36112, 1967. 

Westbach, E. 0. An analysis of a specialized vs. generalized approach to 
pilot training. M-3558:-7-U, W5232a, Research study, AUL/LRX, Max- 
well AFB, AL 36112, 1972. 

Whalen, J. P. Sensitivity training as an aid to air force leadership develop 
ment. M. B. A. Thesis, Columbus, Ohio: Ohio State University, 1967. 

Whitcomb, M. A. (Ed.) Recent developments in vision research. Washington, 
D. C: National Academy of Sciences — National Research Council, 1966. 

Whitlock, G. H. Application of the psychophysical law to performance 
evaluation. Journal of Applied Psychology , 1963, £7, 15-23. 



219 



66 



ERIC 



Why are public housing directors retiring or being fired? Journal of 
Housing , 1971, 28 (2), 86-88. 

Wicker t, F. R. & McFarland, D. E. (Eds.) Measuring executive effective- 
ness . New York: Appleton-Century-Crof ts , 1967 . 

Widdison, K. A. & Skipper, J. K. The use of an interval scale technique 
with occupational prestige rankings. A Paper Presented at the 
Annual Meeting of the American Sociological Association, Denver, 
Colorado, August 30-September 2, 1971. 

Wilber, G. L. Determinants of poverty. A Paper Prepared for the Joint 
Session of the American Sociological Society, Denver, Colorado, 
August 1971. 

Wilcox, L. D. & Brooks, R. M. Social indicators: An alternative approach 
for future research. Ames, Iowa: Iowa State University, 1971. 

Wilcox, L. D. & Brooks, R. M. Toward the development of social indicators 
for policy planning. A Paper Presented at the Annual Meeting of 
the Ohio Valley Sociological Society, Cleveland, April 1971. 

Wilcox, L. D., Brooks, R. M. , Beal, G. M., & Klonglan, G. E. Social indica- 
tors and societal monitoring , an annotated bibliography . San Fran- 
cisco: Jossey-Bass, Inc., Pub., 1972. 

Wilcox, L. D. , Brooks, R. M. , Beal, G. M., & Klonglan, G. E. Social indica- 
tors: Recent trends and selected bibliography. Sociological 
Inquiry , 1972, 42 (1), 37-50. 

Wilcox, R. B. The application of feedback control techniques to organiza- 
tional systems. IRE Transactions on Automatic Control , Vol. AC-7, 
1962, March, 55-67. 

Wilensky, K. L. Organizational intelligence : Knowledge and policy in 
government and industry . New York: Basic Books, 1967. 

Wilensky, H. L. Intelligence in industry: The uses and abuses of experts. 
The Annals of the American Academy of Political and Social Science , 
1970, 388, 46-58. 

Wilkins, W. The managerial grid seminar: Its potential for use by air 

command and staff college. M-*35582-7-U, W6851m, Research study no. 
1390-70, AUL/LRX, Maxwell AFB, AL 36112, 1971. 



250 



67 



Willhelm, S. M. The concept of the ecological complex: A critique. The 
America n Journal of Economics ami Sociolog y, 1964, 23, 241-249. 

Williams, C. F. Maintenance officer training: A combined approach. 

35582-7-U, W7223n, Research study, AUl/LRX, Maxwell AFB, AL 36112, 
1972. 

Williams, J. D. The compleat stratej^lst ; Bein?^ a primer on the theory, 
of- games of strategy . New York: McGraw Hill, 1954. 

Williams, Lady. The revolution in industrial training. Sociologi al 
Review Monograph , 1969, j^, September, 89-104. 

Wixiiams, R. M. Individual and group values. The Annals of the American 
Academy of Political and Social S cience , 1967, 37l.> 20-37. 

Wilson, R. J. An investigation of factors essential to selecting and pre- 
paring on-the-job trainers for a post secondary cooperative voca- 
tional-technical education program. Ann Arbor, Michigan: University 
Microfilms, Inc., 1970. (ERIC ED 0A3 770.) 

Wirt, P. M. (Ed.) Future directions in community power research: A 

colloquium. Berkeley, California: The University of California, 
Institute of Government Studies, 1971. (ERIC ED 054 529.) 

Witherspoon, A. D. A behavioral observation index designed to evaluate 
training of correctional officers iu a prison setting. Masters 
Thesis, Auburn University, August 1971. (ERIC ED 058 546.) 

Wood, W. K. A training model for low rent housing management. A training 
kit for instructors. Contract no. llOl with Department of Housing 
and Urban Development, Philadelphia: Stokes, Wood & Associates, 
(undated) 

Woske, D. G. For TAC mission effectiveness — air crew stan/eval. M-32983- 
U, W935f, Professional study no. 4484, AUL/LRX, Maxwell AFB, AL 
36112. 

Wray, R. 0. Big stick II, a general war exercise. M-32983-U, W94326, 
Professional study, AUL/LRX, Maxwell APB, AL 36112, 1972. 

Yost, M. & Monnin, F. J. A systems approach to the development of an eval- 
uation system for ESEA Title III projects. Bethesda, Maryland: ERIC 
Document Reproduction Service, 1969. (ERIC ED 047 356.) 



25i 



68 



ERIC 



Zachert, M. J. K. & Pantelides, V. S. A computer-assisted sequential in- 
basket technique. Educational Technology , 1971, December, 44-45, 

Zachert, M. J. K. & P< ^elides, V. S, SIBE: A sequential in-basket 

exercise technique. Tallahassee, Florida: Florida State University, 
1971. (ERIC ED 056 469.) 

Zade.h, L. A. On the definition of adaptivity. Proceedings of the IEEE , 
1963, 51 (March), 469-470. 

Zolnerowich, P. The development of a religious education teacher corps. 
358.34, A2988S, Student Report no. 22, AUL/LRX, Maxwell AFB, AL 
36112, 1968. 

Zucker, C. Environmental measurement. New York: City College of the 

City University of New York, School of Architecture and Environmental 
Studies, Urban Research Group, 19/1. 

Zwiebach, Democratic theory and coranunity conurol: Community issues, 

March 1969. Flushing^ New York: City University of New York, Insti- 
tute for Community Studies, March 1959. (ERIC ED 050 205.) 



ADDENDUM 



American Psychological Association, Inc. Publication manual of the 

American Psychological Association. 1967 Revision, Washington, 
D. C: American Psychological Association, Inc., 1967. 

Ammerman, H. L. Development of procedures for deriving training objectives 

for junior officer jobs. Technical Report no. 66-3, Washington, D. C: 
The George Washington University, Human Resources Research Office, 
May 1966. 

Anastasi, A. Psychological Testing . Second Edition, New York: Macmillan 
1954. 

Anastasi, A. Psychological Testing . Third Edition, New York: Macmillan. 
1968. 

Baker, E. L. The technology of instructional development. In R. M. W. 
Travers (Ed.), Second handbook of research on teachin g. Chicago: 
Rand McNally, 1*9731 

Briggs, L. J. et al. Instructional meela . Monograph no. 2, Pittsburgh: 
American Institutes for Research, 1967. 

Bloom, B. S., Engelhart, M. D., Furst, E. J., Hill, W. H. , Krathwohl, D. R. 
Taxonomy of Educational Ob.lectives . The Classification of Educational 
.52ili> Handbook I: Cognitive Domain" New York: David McKay Company, 
Inc., 1956. 

Bloom, B. S., Haskings, J. T., and Madaus, G. F. Handbook on Formative and 
Summative Evaluation of Student Learning. New York: McGraw-Hill 
1971: 

Borus, M. E. and Tash, W. R. Measuring the Impact of Manpower Programs . 
A Primer . Ann Arbor, Michigan: Institute of Labor and Industrial 
Relations, The University of Michigan, Wayne State University, 
November 1970. 

Brewer, G. D. Politicians, Bureaucrats and the Consultant . New York: Basic 
Books, 1973^ ' 

Bush, W. J., Gregg, K. D., Smith, E. A., and McBride, C. B. Some interactions 
between individual differences and modes of instruction. AMRL-TR-65- 
228, Aerospace Medical Research Laboratories, Aerospace Medical Division, 
Air Force Systems Command, Wright -Patterson AFB, Ohio, December 1965. 



253 



ERIC 



2 



Carter, L. F. Exercising the executive decision-making function in large 
systems. In Glaser R. (Ed.)> Training Research and Education , 
Pittsburg: University of Pittsburg Press, 1962. 

Clark, H. E. and Sloan, H. S. Classrooms in the Factories, New York: 
University Press, 1958. 

Cronbach, L. J. Course improvement through evaluation. Teacher College 
Record. 64. (May), 1963. 

Cronbach, L. J. Essentials of Psychological Testing. New York: Harper 
and Row, Pub., 1970. 

Cronbach, L. J., and Gleser, G. C. Psychological Tests and Personnel 
Decisions . Second Edition, Urbana, Illinois: University of 
Illinois Press, 1965. 

Dale^ E. and Chall, J. A formula for predicting readability. Educational 
Research Bulletin , 1948, 27^, 11-20. 

Dale, E. and Chall, J. A formula for predicting readability: Instructions. 
Educational Research Bulletin , 1948, _27, 37-54. 

Dossett, W. F. Likelihood ratio analysis (LRA): A new technique of 
information-variance analysis . HFR Special Report. Goleta, 
California: Human Factors Research, Inc., 1965. 

Duncan, 0. D. Toward social reporting: next steps. Social Science 

Frontiers, Volume 2 . New York: Russell Sage Foundations, 1972. 

Evaluation for local manpower program planners. Background Notes for a 
Presentation at City Manpower Planners Meeting, 77-Attachment 4, 
Louisville, Kentucky, May 24, 1971. 

i'attu, N. A. A catalog of trouble shooting tests . Research Report no. 1, 
Bloomington, Indiana: Indiana University, Institute of Educational 
Research, December 1956 • 

Flanagan, J. C. The critical incident technique. Psychological Bulletin , 
1954, 51, 327-358. 

Flanagan, J. C. and Miller, R. B. The performance record: Handbook for 
foremen and supervisors of hourly employees . Chicago : Science 
Research Associates, 1955. (a) 

Flanagan, J. C. and Milier, R. B. The performance record: Handbook for 
supervisors of non supervisory salaried employees . Chicago : 
Science Research Associates, 1955. fb) 

25 4 



3 



Flanagan, J. C. and Miller, R. B. The performance record: Handbook for 
superintendents and general foremen , Chicago: Science Research 
Associates, 1955. (c) 

Flanagan, J. C. and Miller, R. B. The performance record: Administrators 
manual , Chicago: Science Research Associates, 1955. fd) 

Fleishman, E. A. The description and prediction of perceptual-motor skill 
learning. In Glaser, R. (Ed.), Training Research and Education . 
Pittsburg: University of Pittsburg Press, 1962. 

Frederiksen, N. Proficiency tests for training evaluation. In Glaser, R. 
(Ed.), Training Research and Education . Pittsburg: University of 
Pittsburg Press, 1962. 

Gagne, R. M. (Ed.) Psychological principles in system development . New 
York: Holt, Rineh?rt and Winston, 1962. 

Gam, H. A. Planning, allocation and control in a decentralized, comprehensive 
manpower program . Washington, D. C. : Urban Institute, June 1971. 

Ghiselli, E. W. and Brown, C. W. Personnel and industrial psychology. New 
York: McGraw-Hill, 1955. 

Glaser, R. M. and Nitko, A. J. Measurement in learning and instruction. 

In R. L. Thorndike (Ed.), Educational measurement . Washington, D. C: 
American Council on Education, 1971. 

Glennan, T. K. Evaluating federal manpower programs: Notes and observations . 
Memorandum RM 5743-OEO, Santa Monica, California: Rand Corporation, 
September 1969. 

Gregg Division/McGraw-Hill Book Company and the Center for Vocational and 

Technical Education at the Ohio State University. Writing performance 
goals: Strategy and prototypes. New York: Gregg McGraw-Hill. 

Guilford, J. P. Fundamental statistics in psychology and education . New 
York: McGraw-Hill Book Company, 1965. 

Guion, R. M. Personnel testing . New York: McGraw-Hill, 1965. 

Henriot, P. J. Political aspects of social indicators: Implications for 
research. Social Science Frontiers, Volume 4 . New York: Russell 
Sage Foundations, 1972. 

Human Resources Research Office. Interim bibliography of publications. 
Alexandria, Virginia: The George Washington University, July 1 — 
December 31, 1967. 



255 



ERIC 



Human Resources Research Office. Use of job and task analysis in training. 
Professional Paper no. 1-69, Washington, D. C. : The George 
Washington University, January 1969. 

Insko, C. A. Theories of attitude change. New York: Appletou-Century- 
Crofts, 1967. 

Jones, M. V. and Flax, M. J. The quality of life in metropolitan 

Washington, D. C: Some statistical benchmarks. Washington, D. C: 
The Urban Institute, March 1970. 

Kopkind, A. The future-planners. The New Republic , 1967, 156 , (8), 19-23. 

Lippitt, G. L. Emerging trends in training and development. Training and 
Development Journal , 1969, 23 (2), 2-3. 

Mackie, R. R., Wilson, C. C, Buckner, D. N.,DeLoach, W. D., Dossett, W. F. 
and Ross, E. B. Practical performance test batteries for electrician 
mates and radio men. (Developed in conjunction with a manual for use 
the preparation and administration of practical performance tests for 
the Bureau of Naval Personnel (Pers 15), U. S. Navy and the Personnel 
and Training Branch, Psychological Sciences Division, Office of Naval 
Research.) Contract Nonr 876 (00). Los Angeles: Management and 
Marketing Research Corporation, 1953. 

Mager, R. F. Developing attitude toward learning . Belmont, California: 
Fearon Pub. /Lear Siegler, Inc., 1968. 

Mager, R. F. and Beach, K. M. , Jr. Developing vocational instruction . 
Palo Alto, California: Fearon Pub., 1967. 

McClelland, W. A. The process of effecting change. Professional Paper 
no. 32-68, Washington, D. C. : The George Washington University, 
Human Resources Research Office, October 1968. 

McGeoch, J. A. and Irion, A. L. The psychology of bureau learning . Second 
Edition. New York: Longman, 1952. 

Miller, R. B. Analysis and specification of behavior for training. In 
Glaser, R. (Ed.), Training Research and Education . Pittsburg: 
University of Pittsburg Press, 1962. 

Nitko, A. J. Criterion-referenced testing in the context of instruction. 

Pittsburg: University of Pittsburg, Learning Research and Development 
Center, 1972. 



Scanlon, J. W. , Buchanan, G.^ Nay, J., and Wholey, J. An evaluation 
system to support planning, allocation and control in a 
decentralized, comprehensive manpower program.. A Final Report, 
Washington, D. C: Urban Institute, March 1971. 

Shinn, A. M. Measuring the utility of housing: demonstrating a 

methodological approach. Social Science Quarterly , 1971, June, 
88-102. (b) 

Smith, R. G. An annotated bibliography on the design of instructional 
systems. Technical Report no. 67-5, Alexandria, Virginia: The 
George Washington University, Human Resources Research Office, 
May, 1967. 

Smith, R. G. The engineering of educational and training systems . 
Lexington, Massachusetts: Heath Lexington Books, 1971. 

Standlee, L. S., Popham, W. J., and Fattu, N. A. A review of trouble- 
shooting research. Research Report no. 3, Bloomington, Indiana: 
Indiana University, Institute of Educational Research, December 
1956. 

Steinmetz, C. S. The evolution of training. In R. L. Craig and L. R. 
Bit t el (Eds.), Training and development handbook . New Ydrk: 
McGraw-Hill, 1967. 

Stufflebeam, D. I., Foley, W. J., Gephart, W. J., Guba, E. G., Hammand, 
R. I., Merriman, H. 0., and Provus, M. M. Educational evaluation 
and decision making . Itasca, Illinois: F. E. Peacock Publishers, 
Inc., 1971. 

Tarleton, H. R. Mobilization augmentees in ICBM operations. M-32983-U, 
T188m, professional study. AUL/LRX, Maxwell AFB, AL 36112, 1972. 

Taylor, E. W. The influence of rating scale construction upon the 
characteristics of ratings rendered. Cleveland: Personnel 
Research and Development Corporation, 1958. 

Tracey, W. R. Evaluating training and development systems. New York: 
American Management Association, Inc., 1968. 

Travers, R. M. W. A study of the relationship of psychological research 
to educational pratice. In Giaser, R. (Ed.), Training research 
and education . Pittsburg: University of Pittsburg Press, 1962. 

Triandis, H. C. Attitude and attitude change. New York: John Wiley 
and Sons , Inc . , 1971. 



257 



6 



Turabian, K. L. A manual for writers of term papers, theses, and 

dissertations . Third Edition, Revised, Chicago: The University 
of Chicago Press, 1967, 

Tyler, R, W, Analysis of the Purpose, Pattern, Scope and Structure of 
the Officer Education Program of Air University, Maxwell AFB, 
Alabama: Officer Education Research Laboratory, Air Force Personnel 
and Training Research and Development Command, May 1955, 

United States Air Force, Office of Information, Air training command 
fact sheet , September 1972. 

U. S, Civil Service Commission, Bureau of Training, Studies and reports 
relating co training and ed!?cation . Pamphlet T-6, FY1969, 
Washington, D, C, March 1970, CSI,48:T-6/2, 

U. S, Civil Service Commission, Bureau of Training, Training evaluation : 
A guide to its planning, development and use in agency training 
courses. Training Systems and Technology Series: no, IV, May 19 71, 

Veldman, D. J. Fortran Programming for The Behavioral Sciences , New York: 
Holt, Reinhart and Winston, 1967, 

Webb, E. J., Campbell, D. T., Schwartz, R. D,, and Sechrist, L, Unobstrusive 
Measures: Nonreactive Research in the Social Sciences , Chicago: 
Rand McNally, 1966. 

Wenig, R. E. and Wolansky, W, D, Review and synthesis of literature on 
job training in industry , Columbus, Ohio: ERIC Clearinghouse on 
Vocational and Technical Education, 19 72, 

Wilcox, L, D., Brooks, R. M, , Beal, G, M, , and Klonglan, G, E, Social 
indicators and societal monitoring, an annotat ed i^liography , 
San Francisco: Jossey-Bass Inc,, Pub., 197'^„ 

Wilson, C, L, On-the-job and operational criteria. lu Glaser, R, (Ed.), 

Training research and education , Pittsburg: University of Pittsburg 
Press, 1962. 



25S 



Appendix A 

Outline of Underlying Assumptions for DOES Development 
1.41 Sampling Assumptions 

1.411 Sampling Assumptions/Population 

1.4111 Training Population 

1.41111 Trainees are representative of the class of 
employees for whom the training is intended 
(e.g., sex, size of community) 

1.4112 Training Instructors 

1.41121 T.I. are representative of all such T.I. 
who provide such training 

1.4113 Decision Makers 

1.41131 D.M. are representative of all such D.M. 
who use decision informatio.; 

1.411311 Middle Management 

1.411312 Top Management 

1.4114 a tors 

1. rl Evaluatora who use this system are represen- 
tative of evaluators who value and employ a 
decision orientation to evaluation 

1.4115 On the Job Co-workers 

1.41151 Co-workers are representative of an occupa- 
tional class with whom trainees interact 

1.411511 Peers 

1.411512 Subordinates 

1.411513 Superiors 
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1.4116 Conanunity members who provide impact data 

1.41161 Are representative of class of people 
being sampled 

1.412 Work Sampling 

1.4121 Skill performances to which training is being given 
is important and representative of the class of 
behavior vrfiich an occupation should possess. 

1.413 Time Sampling 

1.4131 Distance from training 

1 .41311 Community impact 

First sample = 3 to 6 months after treatment 
Second sample = 15 to 18 months after treatment 

1.4132 Amount of time sampled 

1.41321 Job performance 

1.413221 Behavior 8 to 24 hours/person 

1.413222 Records 

1 year prior to training/time 
samples and training to visitation 

1.41322 Community impact (size of temporal unit) 

1.413221 Records scan 1 year observation 
period 

1.413222 Direct observation 5 days/obser- 
vation period 

1.413223 Interviews — 5 days /observation period 
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1.42 Treatment Assumptionb 

1.421 Treatment Delivery (skills) 

1.4211 The treatment will produce the skill behaviors it has 
specified as outcomes 

1.4212 There will be a transfer of treatment effects to 
job performance 

1.4213 There will be a transfer of job performance effects in 
terms of indicators of community impact related to 
training. (There will be a transfer of effects from 
treatment through job performances to a community 
impact indicator . ) 

1.422 Additional Treatment Effects 

1.4221 Awareness will increase 

1.42211 Cognitive outcomes 

1.42212 Affective outcomes 

1.4222 Attitudes will change 

1.4223 Knowledge increments will occur 
1.43 Measurement Assumpt:_ons 

1.431 General Measurement Assumptions 

1.4311 Instruments used to measure effects of training, job 
performance, and community impact will be non-three.teniag 

1 .43111 Unobtrusive measures /Observations techniques 

1.43112 Simulation/Problem solving activities will 
be used 

1.4312 The evaluation system will make use of natural data 
(historical records, process records) where possible. 
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1.4313 Trained observers /interviewers are the only authorized 
data gatherers. 

1.4314 Where possible data will be collected in its natural 
setting • 

1.4315 Longitudinal and cross-sectional data will be collected 
where possible 

1.4316 All ratings will be made by trained raters. 

1.4317 Evaluator will be allowed to place trained observers 

in the system whenever and wherever necessary (training 
package development, training situation, job environ- 
ment, and community) 

1.4318 Validity and reliability will be determined for all 
data collection devices before those devices are used 

to obtain evaluation data on treatment, job performance, 
and community impact. 
1.432 Training Materials Measurement Assumptions 

1.4321 Reliability of rating evaluation of training materials 

1.43211 Interrater agreement 

1.43212 Intrarater agreement 

1.4322 Quality of package can be developed 
1.43221 Expert analysis 

1.432211 Format 

1.432212 Content adequacy/relevancy/ 
internal consistency 

1.432213 Intelligibility 
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1.43222 Reliability of experts' analyses will be 
determined 

1.43223 Ratings can be scaled on a quality/index 
basis 

1.4323 There is face validity of the package it is addressed 
to the needs /problems for which the contract was let. 

1.4324 Indices of reliability, and validity can be developed 
and will be functionally useful for making decisions. 

1.433 Training Process (Treatment) Measurement Assumptions 

1.4331 Not everything can be measured 

1.4332 We can measure the extent to which objectives of 
training are being met. 

1*4333 We can defir.„ variables affecting successful/ 

unsuccessful achievement of training objectives 
(diagnostic information) /Variables = 



1. 


Simulation 


2. 


Workshop 


3. 


Seminar 


4. 


Lecture 


5. 


Self-study 


6. 


Trainees 


7. 


Instructors 


8. 


Behavioral Objectives 


9. 


Materials 


10. 


Self -evaluation 


11. 


Monitoring 


12. 


Roles 


13. 


Interaction 
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14. Setting 

15. Performances 

16. Attitude 

17. Length of training 

18. Duration 

19. Exit/Entry Level/Behavior Skills 

20. Skill training 

2 1 . Knowledge inc rement 

22. Awareness 

a) affective 

b) cognitive 

1.434 Job Performance Measurement Assumptions 

1.4341 Variables for measuring job performance can be 
found or developed that are directly related to 
skills developed in training (performance measure- 
ment devices and ratings) 

1.4342 Job performance measures will provide valid criteria 
for the evaluation of transfer training effects 

1.4343 Data from "job records" can be obtained which are 
related to skills developed in training. (Before 
and after treatment.) 

1.435 Community Impact Measurement Assumptions 

1.4351 Indicators of "community impact" can be related to 

training objectives and that the iutervening process 
of job performances mediates between objectives of 
training and bring about changes in the community. 
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1.44 Analysis Assumptions 

1»441 Training Materials 

1.4411 Ratings of training ttiaterials will provide goodness/ 
badness descriptions for the total package and aub- 
package elements 

1.44111 The standard for rating is valid and reliable 

1.44112 The standard is used for comparative purposes 
by judges to yield descriptions 

1.442 Training Materials Implementation Analysis Assumptions 

1.4421 Analyses will be performed to describe and explain 
(account for statistically) training effects 

1.4422 Analyses to describe and explain training effects 
will be used to 

a) provide for validation/eelection and refinement 
of "package analysis criteria" 

b) predict job performance and impact criteria 

1.443 Job Performance Analysis Assumptions 

1.4431 Comparative analyses Will be conducted to compare end 
of training performance with job performance 

1.4432 Explanatory analyses Will be conducted to determine 
presence or absence of transfer of training effects 
to job performance 

1.444 Conmunity Impact Analysis Assumptions 

1,4441 Impact analyses will «lescribe observable effects in 

the conmunity directly or indirectly related to training 
and job performance 
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1.4442 Impact analyses could be uoed to coir»i 5re observable 
effects of training with control communities 

1.4443 Impact analysis will he used to eixplaln the presence 
or absence of obser^/able effects on the community 
related directly/indirectly to training and job 
performan:ie . 

1.445 General Analysis Assumptions 

1.4451 Predictive analyses might be performed to ascertain 
the relationship between critical variables related 
to training, job performance, and community impact 
and hypothesized future response capability. 

1.4452 Change score analyses will not be conducted; control 
group comparative analyses provide the best alternatives 
to describing and explaining effects of training, job 
performance and community impact. 

1.4453 Analyses will be performed on only clearly demonstrable 
phenomena . 

1.446 Statistical Analysis Assumptions 

1.4461 The types of data available and the questions to be 
answered control the statistical analyses to be 
employed. 

1.4461 The sampling plan derives from data 
available, questions to be answered, 
constraints, and evaluation design; the 
above determine data o be collected. 
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1.4462 Statistical analysis to be used may include 
the following: 

a) Correlational analysis 

b) Analysis of variance 

c) Analysis of covariance 

d) Regression analysis 

e) Multiple regression analysis 

f) Distribution free analysis 

3.1 Preparation of Reports 

3.11 Report Audiences — Reports of evaluation will be directed to 

3.111 Decision makers 

Local (top, middle, technical) 
State (top, middle, technical) 
National (top, middle, technical) 

3.112 Developers 

3 . 1 13 Evaluator s 

3.12 Reporting Levels — Types of reports 

3.121 Technical Reports 

3.122 Technical Report Abstract 

3.123 Decision Information Summary 
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APPENDIX B 
Training or Information Sites Visited 
by the Search &^ Analysis Team 



The individuals who gave generously of their time and 
shared their thinking on training and evaluation are: 



Floyd Hacker 

Administrator of D.P.S, Training 
Texas Department of Public Safety 
5805 North Lamar 
Austin, Texas 
September 11, 1972 

T Sgt. Ronald Moon 

Head, Manager Training School 

Bergs vrom Air Force Base 

Bastrop Road 

Austin, Texas 

September 14, 1972 

Warren Adkins 

Manpower Training Program 

Texas Employment Commission 

P. 0. Box 160 

Austin, Texas 

September 18, 1972 

C, Robert Woods 

D. P.S, Programmed Courses 
Plannet Association 

900 West Koenig Lane #117 
Austin, Texas 
September 22, 1972 

T Sgt, Vozza 

N.C.O. School 

Bergstrom Air Force Base 

Bastrop Road 

Austin, Texas 

September 25, 1972 
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Maj. Gen. Gordon F. Blood and 
Lt . Col . James Boyd 
12th Air Force 

Bergs troTT Air Force Base, Texas 
September 26, 19 72 

Lynn Anderson 

Dean of the Office of Conference and Training 

LBJ School of Public Affairs 

Sid Richardson Hall 

Unit 3, Room 3.13 

Austin, Texas 

September 27, 1972 

David Spurgin 

Texas State Job Training Program 
LBJ School of Public Affairs 
Sid Richardson Hall 
Unit 5, Room 3.13 
Austin, Texas 
September 29, 1972 

Bill Wettington 
Safeway District Manager 
6101 North Hampton 
Austin, Texas 
October 3, 1972 

Joe Riordian 

Public Relations Manager 
Southwestern Bell Telephone Company 
Austin District 
909 Colorado 
Austin, Texas 
October 4, 1972 

Tom Phillips 

Sears Introductory Bid Sales 
Personnel Training" 
Hancock Shopp^nt .ic 
Austin, Texas 
October 5 and 12 » ly72 

Malcolm McClinchie 
Training Officer 
IBM 

11400 FM Road 1325 
Austin, Texas 
October 25, 1972 

Al Jones 

Coordinator of Manpower Training 
Texas Education Agency 
Brown Building, Room 209 
Austin, Texas 
October 30, 1972 
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APPENDIX C 



LIBRARY REFERENCE SEARCHED 



Encyclopedia Search 



Encyclopedia of Educational Research 

Dates Searched ; 1959 (fourth edition, pp. 1-124) 

Major Descriptors Used ; supervision, housing maintenance 

management, training, on the job 
performance, community impact 

Minor Descriptors Used ; education, socialization, community 

development, political socialization 

Location ; Education and Psychology Library 
Catalog Search 



Monthly Catalog 

Dates Searched ; (1) January, 1970 — December, 1970; (2) 
January, 1971 — September, 1972 

Major Descriptors ; (1) supervision, housing maintenance 

management, training, cn the job per- 
formance, community impact; (2) eval- 
uation, community impact, housing main- 
tenance management, supervision 

Minor Descriptors Used ; (1) job education, decision- 
making , overnment training , 
mili^^ry raining , manpower , 
job .orp; (2) management, admin- 
istrative management, training, 
miciule management, job descriptors, 
city administration 
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Location : Documents Library, Main Building, 3rd floor; 

also. State Library Building, 12th and Brazos 
Streets, main reading room. 



General Files Search 



Location: Texas Department of Conununity Affairs, Sam 
Houston Building, 14th and San Jacinto Sts,, 



Materials Searched: 



training, evaluation, city manage- 
ment, housing management, community 
impact, low-rent housing, training 
packets, local government. 



ERIC~C.I. J.E. 



Index Search 



Dates Searched: 1969—1972 



Major Descriptors Used : 



supervision, housing management 
training, job performance, community 
impact, evaluation, taxonomies of 
training objectives. 



Minor Descriptors Used : 



action programs, action research, 
administrative personnel, admin- 
istrative education, affective ob- 
jectives, community involvement, 
community analysis , decision-making, 
evaluation, housing, management ed- 
ucation, job analysis J planning, 
program administration, systems 
analysis, training, validity, envir- 
onmental criteria, social characteristics 
community change, community study, 
social influences, social action, 
community services, evaluation, state 
programs, training objectives, taxonomy, 
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job skills, task performance, 
skill analysis, objectives, per- 
formance criteria, evaluation methods, 
site analysis, participant satisfaction, 
program evaluation, cost effectiveness, 
supervisory training, transfer of 
training 

Location ; Research and Development Library, Education 
and Psychology Library. 

ERIC— R,I.E, 

Dates Searched : 1969—1972 

Major Descriptors Used : supervision, housing management, 

training, job performance, community 
impact, evaluation, taxonomies of 
training objectives. 

Minor Descriptors Used : action programs, action research, 

administrative personnel, administra- 
tor evaluation, affective objectives, 
community involvement, community 
analysis, decision-making, evaluation, 
housing, management evaluation, job 
analysis, planning, program administration, 
systems analysis, training, validity, 
environmental criteria, social charac- 
teristics, community change, community 
study, social influences, social action, 
community services, evaluation, sLace 
programs, training objectives, taxonomy, 
job skills, task performance, skill 
analysis, objectives, performance cri- 
teria, evaluation methods, site analysis, 
participant satisfaction, program eval- 
uation, cost effectiveness, supervisory 
training, transfer of training. 

Location: Education and Psychology Library 



272 

o 

ERIC 



- 4 - 



Index of Business Periodicals 

Dates Searched ; January, 1971 — December, 1971 

Major Descriptors Used ; supervision, housing maintenance 

management, training, on the job per- 
formance, community impact 

Minor Descriptors Used ; management training, business man- 
agement and supervision, personnel 
management, housing management- om- 
munity studies, evaluation tecii. ques, 
administrative education, theories of 
decision-making 

Location : Business Administration Library, B.E.B. 



Psychological Abstracts 

Dates Searched ; (1) June, 1968 — December, 1969; (2) January, 
1970— October, 1972 

Major Descriptors Used ; (1) training, on the job per- 

formrace, community impact; 
(2) supervision and housing 
maintenance management, training, 
on the job performance, community 
impact 

Minor Descriptors Used ; (1) management, evaluation; 

(2) personnel management, decision- 
making, education, evaluation, com- 
munity impact, social indicators, 
supervisors, training methods^ man- 
agement, housing, methodology 

Location; Education and Psychology Library 



Sociological Abstracts 

Dates Searched ; January, 1970 — December, 1971; February- 
August, 1972 
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Major Descriptors Used ; community impact, community change, 

evaluation, social indicators, train- 
ing 

Minor Descriptors Used : job training, evaluation, impact, 

community, community development, 
community rela io..^, cities, city 
planning, inde::, indicators, social, 
social action, social change, social 
planning, social research. 

Location ; Main Library, Social Science Reference Room 

U*S. Government Research and Development Reports 

Dates Searched: January, 1970 — Decembex ^ 1970 

Major Descriptors Used ; supervision, training, on the 

job performance 

Minor Descriptors Used : evaluation, government training, 

supervisory training, management 
training, technical training 

Location: Documents Library, 3rd floor. Main Building 

Air University Abstracts of Research Reports 

Dates Searched ; 1968, 1970, 1971, 1972 

Descriptors ; Specific descriptot ^ qjie iiidexed under 
general headings. Therefore, the entire 
subject index was searched for community 
impact, job performance, training, and 
evaluation. 

Location; Air Force ROTC Office, ROTC Building, 

Air University Library, Maxwell Air Force 
Base, Alabama 
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Journal Search 



Canadian Public Administration 

Dates Searched ; Summer — Fall, 1972 

Major Descriptors Used ; housing maintenance and management 

community impact 

Location; Social Science Reference Room 



Journal of Applied Psychology 

Dates Searched ; January, 1969 — January, 1970 

Maj^or D:;:s c.riptors Used : community impact, supervision and 

housing maintenance management 

Minor Descriptors Used : management, training, evaluation 

Location ; Education and Psychology Library 

Journal of Housing 

Dates S earched ; January, 1970 — December, 1972 

Major Descriptors Used ; supervision and housing management, 

training, on the job performance, 
community impact 

Minor Descriptors Used ; government training, government on 

the job performance, government eval- 
uations of; training, on the job per- 
formance, community impact, building 
supervisors, decision-making, social 
indicators 

Location; Archi lecture Library (Architecture Building) 



International Journal of Politics 



EKLC 



Dates Searched ; January, 1970 — December, 1971 
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Major Descripto rs Used : communitv impact 

Minor Descri ptors Used : job training, eval-u^u ^.vi, impact, 

community, coramunra:- development, 
community relations 

Location : Main Library, Social Science Reference Room 



Training and D3velopment Journal 

Dates Searched : (1) January, 1968 — May, 1963; (2) June, 1968 
— J^^ u- 1969 

Maj or Descriptors U::^ ^) community impact, job performance, 

ciTaining, supervision housing mainten- 
ance management; (2) community impact, 
supervision and housing maintenance 
management 

Min or Descriptors Used ^ (1) job training, evaluation, impact, 

community, management training, educa- 
tion, military training, military edu- 
cation, community development, com- 
munity relations; (2) management, 
training, evaluation 

Location ; Business Library: B.E.B. 



Public A dm 1 a . 1 .s t r a t i o n : Journal of the Royal Institute of Public 
Aan) i n i s n r a t io n 

Dates Searched : Summer, 1972 — Autumn, 1972 

Major Descriptors Used : housing maintenance management, community 

impact 

Location : Main Library, Social Science Reference Room 



Urban Research News (a newsletter) 

Dates Searched : January 5, 1970 — January 18, 1971 
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Major Descriptors Used : housing managemepL 

Minor Descriptors Used : housing studies, low-rent housing, 

urban housing 

Location: Main Library, Social Science Reference Room 



Bibliography Search 



Beal, George M, , Brooks, Ralph M. , Wilcox, Leslie D, , and 
Klonglan, Gerald E. Social indicators : bibliography 
1_ (Report no. 92). Ames, Iowa: Iowa State University, 
Department of Sociology and Anthropology, 1971. 

Location: Main Library, Social Science Reference Room 



Wilcox, L.D., Brooks, R.M. , Beal, G.M., and Klonglan, G^E. 

Social indicators and societal monitoring : an annotated 
bibliography . San Francisco: Jossey-Bass, Inc., Pub- 
lishers, 1972. 

Des^iriptors? indicators, att'.tuca indicators, economic 

indicators, citizen partici ation indicators, 
electoral participation indicators, political 
indicators, social effects indicators, social 
impact indicators, social mobility indicators, 
social statistics indicators, values indicators 

Location: I 
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Corr^jpondence 



1. Director, Human Resources 
ilesearch Office 

300 North Washington St. 
Alexandria, Virginia 22314 

2. Mr. Glen H. McLaughlin, Chief 
Personnel and Staff Services 
Texas D.P.S. 

Box 4087 N.A.S. 
Austin, Texas 

3. Ma j . General Gordon F. Blood, Commander 
12 Air Force 

Bergstrom Air Force Base, Texas 78743 

4. Director, International Association of Chiefs of Police 
1319 18th St., N.W. 

Washington, D.C. 20036 

5. The Office of the Governor 
State of Connecticut 

210 Capitol Ave. 

Hartford, Connecticut 01615 

6. The Office of the Governor 
State of New York 

Eagle State and Washington 
Albany, N. 12224 

7. Office of the Governor 
State of Ohio 

State House 
Columbus, Ohio 43215 

8. Office of the Governor 
State of Illinois 
State Capitol Complex 
Springfield, Illinois 61106 
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9. Office of the Governor 
State of NorlL Dakota 
Bismark, ^^'^rth Dakota 58501 

10. Office of the Governor 
State of Michigan 
Lansing, Michigan 48933 

11. Office of the Governor 
State of Wisconsin 
Capitol Square 
Madison, Wisconsin 53702 

12* Officii of the Governor 
State of Oregon 
Salem, Oregon 97310 

13. Office of the Governor 
State of Washington 
Olympia, Washington 98504 

14. Office of the Governor 
State of California 
Sacramento, California 95814 

15. Office of the Governor 
State of Colorado 

200 East Colfax 
Denver, Colorado 80903 

16. Xhe Office in Charge of Special Police Training 
Uni /ersity of Kentucky at Louisville 
Lotiir-'ille, Kentucky 40208 

17. The Office in Charge of Special Police Training 
Noithwestern University 

Evx^r^sr.cn:, Ix^.inois 60201 

18* The Office ii. Cnarge of Special Police Training 
Central Missouri St^te College 
Warrensburg, Missouri 64093 
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Mr. Bill Wetting ton 
Safewa;^:- District Manager 
6101 Korth Hampton 
Austin, Texas 78723 

20. Mr. Tom Phillips 
Personnel Director, Sears 
Hancock Shopping Center 
Austin, Texas 78751 

21. Mr. Joe Bacon, Manager 
Ramada Inn 

5656 N. Interregional 
Austin, Texas 

2P. Mr. Joe Riordian 

Public Relations Division 
Southwestern Bell Telephone 
909 Colorado 
Austin, Texas 78701 

23. Mr. Floyd Hacker 
Texas D.P.S. 
P.O. lox 4087 
Austin, Texas 78773 

24. Mr. Lynn Anderson 

Director of Conferences and Training 
L.B.J. School of Public Affairs 
Sid Richardson Hall, Unit 3, Room #3.13 
Austin, Texas 7871?. 

25. Mr. Warren Adkins 

Texas Employment Cominission 
P.O. Box 160 
AuSw/n, Texas 

26. Mr. C. Robert Woods 
Plannet Associates 

900 W. Koenig Lane #117 
Austin, Texas 
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27. Mr. Mel McCIinchie, Project Manager—Personnel 
IBM Corporation 

11400 FM Rd. 1325 
Austin, Texas 78759 

28. AIR 

P,0. Box 1113 

Palo Alto, California 94302 

29. Dr. Terry Eisenberg 
Research Scientist 

Prof. Standards Division of lACP 
11 Firstfield Rd* 
Gaithersburg, Maryland 20760 

30. Mr. Al Jones, Director, Adult Continuing Education 
Texas Education Agency (1£A) 

Brown Bldg., Room #209 
Austin, Texas 

31. Mr. James M. Patton, Chief 
Trnining Dev. Branch 

C: -'lian Personnel ana Training Division 
rix e Sands Missile Range, New Mexico 38001 

Z2. V .rl L. Hofer, Chief 

or ^iu::atlon and Training Branch 
Bureau of Hc^arings and Appeals 
HEW, South ''Idg*, Room //4118 
330 C. Street, S.W. 
Washington, D.C. 20201 

33. Director, Regional Training Center 
Philadelphia Region 

U.; i Civil Service Commission 
U.S. Custom House 
Philadelphia, Pa. 19106 

34. Director, Regional Trainir^ii Cenr.er 
U.S. Civil Service CcJiamiseion 
Atlanta Merchandise Mar': 

240 Peachtree St., N.W. 
Atlanta, Georgia 30303 
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35, Director, Training Division 

Naval Publications and Forms Center 
5801 Taber Ave. 
Philadelphia, Pa. 19120 

36, Mr, Howard G. Baen, Director 
Chicago Regional Training Center 
U.S. Civil Service Connnission 
Main Post Office Bldg., Rcom #1311 
433 West Van Buren St. 

Chicago, Illinois 60G07 

37, Director, Internal Health Programs Systems Cc ter 
Tucson, i ii'.ona 85 "OX 

38, Mr. Curtis Spencer» Director 
Denver Regional Training Center 
U.S. Civil Service Commit ion 
Bldg. 20, Denver Federal Center 
Denver, Colorado 80225 

39, SSRC Center for Coordination of Research on Social 

Indicators 
.1735 Massachusetts Ave., N-,W. 
Washington, D.C. 20036 

40- Superintendent of Documents 

U.S, Government Printing Office 
Washington, D.C. 20402 

41. Personnel P^esearch Labot'atory 
Aerospace Medical Division 
Lackland Air Force Base 
San Antonxo, Texas 78236 

42 U,S. Arniy Engineer School 

rt i^eivoir, Virginia 220^3 

43. Officer of Educational Rrf.search Lab. 
Air Resea:. ch and Develoi itienn Conmiand 
Ms ell Air Force Base 
MaxT'Fell J Cfcilif ornia 36112 
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44. Department of the Navy 
Naval Research Laboratory 
Publications Office 
Washington, D.C. 20390 

45. Personnel Laboratory 
Aeronautical Systems Division 
Randolph Air Force Base 

San Anto: . , Texas 78148 

46* Aerospace Medical Research Laboratories 
Wright -Patterson Air Force Base 
6570th Aerospace Wright-Patterson A.F.B. 
Wright-Patterson, Ohio 

47. Department of the Air Force 
Headquar ter s— Pentagon 
Publications Office 
Washington. D.C. 20330 

48. Department of the Army 
Headquarters 

U.S. Army Recruiting Command 
Advertising and Information 
Hampton, Virginia 23369 

49. Continental Army Command 
Fort Monroe, Virginia 23351 

50. American Institute for Research 
Arastradero St. 

Palo Alto, California 94306 

51. U.S. Army Southeast Signal School 
Fort Gordon, Georgia 30305 

52. ERIC Document Reproduction Service 
P.O. Drawer J 

Bethesda, Maryland 20014 
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j3. U.S. Department of Commerce 

National Technical Information Service 
Springfield, Virginia 22151 

54. Ms. Annette Goldreyer 
1686 Moorings Dr. 
Reston, Virginia 22070 

55 . DDC 

Cameron Station 
Alexandrda, Virginia 22514 

56. Air Force Human Resources Laboratory 
Air Force Systems Command 

Brooks Air Force Base, Texas 78235 

57. "apt. Leslie G. Redmann 
Bldg. 901, Room #316 
Randolph Air Force Base 
San Antonio, Texas 78148 

58. American Educational Research Association 
1126 16th St., N.W« 

Washington, D.C. 20036 

59. Harper and Row, Inc. 
49 East 33rd St. 

New York, N.Y. 10016 

60. Longman Group Ltd. 
74 Grosvenor ^ . 
London WIX Ok.. T^^agland 

61. Public Personnel Association 
' M3 E. 60th St. 

Cliicagc^ IMinois 6063" 

62. Professo. Robert Wooc, Director 

Harvard — KIT Joint Center on Urban Affairs 
66 Church St. 

Cambridge, Massachusetts 02138 
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63. Dr. Angus Campbell, Director 
University of Michigan 
Institute for Social Research 
P.O. Box 1248 
Ann Arbor, Michigan 48106 

64., ^>e Brookings Institute 

1775 Massachusetts Ave., N.W. 
Washington, D.C. 23036 

65. Educational Innovators Press 
Box 13052 

Tucson, Arizona 85711 

66. Russell Sage Foundation 
230 Park Avenue 

New York, N.Y. 10017 

67. Dissemination Office 

Center for the Study of Evaluation 
UCLA Graduate School of Education 
Los Angeles, California 90024 

68. Center fox" urban Studies 
Wayne State University 
Detroit, Michigan 48207 

69. New York State Office of Planning Coordination 
Alb ny. New York 

70. National Opinion Research Center 
University of Chicago 
Chicago, Illinois 606.7 

71. Stanford Research Institute 
Menlo Park, California 

72. Al aquerque Urban Observatory 
Albuquerque, New Mexico 

73. Center for Urban Affairs 
Northwestern University 
Evans ton, Illinois 60201 
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74. Office of the Assistant Secretary for Environment 
and Urban Systems 
U.S. Department of Transportation 
Washington, D.C. 20590 

75 • Urban Research Group 

School of Archit ecture slxhI Urfvuroameiital Studies 
City College of the City University of New York 
New York, N.Y. iOOlO 

76. Wharton School, of Finance and Commerce 
University of Pennsylvania 
Philadelphia, Pennsylvania 19104 

77. American Institutes for Research 
School of Education 
University of Pittsburgh 
Pittsburgh, Pennsylvania 15213 

78. Holt, Rinehart and Winston 
New York, N.Y. 

79. Office of the Chief 

Human Resources Research Organ. :ion 
300 N. Washington St. 
Alexandria, Virginia 22314 

80. School of Education 
Ohio State University 
Columbus, Ohio 43210 

81 o S., Telopment Corporation 

San. ■ .ica^ California 

82 o The Urban Institute 
Washington, D.C. 

83. School of Social W ^rk 
University of Michigan 
Ann Arbor, Michigan 48104 



EKLC 



286 



- 10 - 



84. American Sociological Association 
1722 N. Street, N.W. 
Washington, D.C. 20036 

85. Dr. Byron E, Munson 

Department of Sociology/Anthropology 
Eastern Illinois University 
Charleston, Illinois :i920 

86. Center for Community '^f'udies 
P.O. Box 60 

George Peabody College 
Nashville, Tennessee 37203 

87. Dr, David S. Albert ; 
c/o Praeger Publishers 
Editorial Services 
111 4th Avenue 

New York, N.Y. 10003 

88. American Management Association 
135 West 50th St. 

New York, N.Y. 10020 

S9. Department of Anthropology 
Cornell University 
Ithica, New York 14850 

90. Institute of Labor and Industrial Relations 
The University of Michigan 

Ann Arbor, Michigan 48104 

91. Dr. Paul Street 
Dickey Hall, Room #63D 
T'niversity of Kentucky' 
Lexington, Kentucl y 40506 

92. Science Research Associates 
259 East Erie 

''hicago, Illinois 60611 

1 
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93. T. Geur;;^t. Hcirri;^ 
Psycho l.^^l ;;^oda2 
1330 Cam^'nc del Mar 

Del Mar, C-..X:i.f ornia 90214 

94. Addison-We^ley Publishing Company 
Reading, Massachusetts 01867 

95. Dr. H.S. Bhola 
Indiana University 
Bloomington, Indiana 47401 

96. Spartan Books 

1250 Connecticut Avenue, 
Washington, D.C. 20036 

97. Sylvania i:ilectronic S\ tci^i, 
63 Second Avenue 

Waltham, Massachusetts • '^M54 
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APPENDIX E 

Site Visits Made by Project Team Members 



The people vjho gave generously of their time 
and shared their thinking on training and 
evaluation were: 



M. W. Wachs and William Rogers 
Department of Housing and Urban Development 
Community Planning and Management 
Washington, D, C. 
September 22, 1972 

G. B, Martin, P.P. Mitchell and B. F. Cottingham 
Southwestern Bell Telephone 
San Antonio, Texas 
November 1972 

Nicholas P. Thomas 
.LBJ School of Public Affairs 
Univtirsity of Texas at Austin 
Austin, Texas 
Decemb.er 4, 1972 

Robert L, ^alglish 
San Antonio College 
San Antonio, Texas 
December 18, 1972 

Capt. L. G. Redmann, Col, Lee Young, Maj. F. P. Leuck, 
Col. M. J. Rega, C. C. Cunningham^ Maj. J, J. Blakey, 
Lt, Blaine Lee, CMSgt. C. D. Elliott 
Randolph Air Force Base 
San Antonio, Texas 
January 26, 1973 

W. B, Mansfield, James Musick 
Engineering Extension Division 
T*ixas A&M University 
College Station, Texas 
January 30, 1973 

P. E, Morgette and L. H. Courson 
Personnel Department 
City of Austin 
Austin, Texas 
March 1973 




L. Dalglish and Dick Thiesen 
San Antonio College 
San Antonio, Texas 
March 26, 1973 

Harvey H, Johle, Collas Smelser 

Personnel and Training 

San Antonio Housing Authority 

San Antonio, Texas 

March 26, 1973 

W. E. Driskill 

Occupational Measurement Squadron 
Lackland Air Force Base, Texas 
March 27, 1973 

Roy DeGaugh, J, W, Bowls, W. B, Lecznar 
Joe Ward, Manuel Pina, Col, 0, A, Berthold, 
and Col. David Reingard 
Personnel Research Division 
Lackland Air Force Base, Texas 
March 27, 1973 

Collas Smelser 

San Antonio Housing Authority 
San Antonio, Texas 
April 17, 1973 

Murray Horowitz and Edgar Goff 
Community University Research Associel;- 
Boston College 
Boston, Massachusetts 
April 27-30, 1973 

Richard Lavin 

Merrima-Jc Education Center 
Boston, Massachusetts 
April 27-30, 1973 

Orville Hubbard, Mayor 
City of Dearborn 
Dearborn, Michigan 
May 4, 1973 

Anita Lux 
Deputy Director 
Service Bureau 
City of Dearborn 
Dearborn, Michigan 
May 4, 1973 
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Marilynn Wacker 

Assistant Executive Director 

San Antonio Housing Authority 

San Antonio, Texas 

May 7, 1973 

Harvey Johle, Collas Smelser 
San Antonio Housing Authority 
San Antonio, Texas 
May 7, 1973 

Collas Smelser 

San Antonio Housing Authority 
San Antonio, Texas 
May 16-17, 19 73 

Reynold Blight 
Manpower Analyst 

Los Angeles Connnunity Affairs Manpower 

Planning Survey (CAMPS) 
Los Angeles, California 
May 21, 1973 

Ella Reid Bell 
City Planner 

Los Angeles Community Analysis Bureau (CAB) 
Los Angeles, California 
May 21, 1973 

Marcia Madison and Charlene Groff 
Evaluation Specialists 
City Demonstration Agency 
Evaluation & Information Unit 
Los Ai^[<oles, California 
May 21, 1973 

Peter DeLeon 
Rand Corporation 
Santa Monica, California 
May 22, 1973 

Millicent Cox 

Urban Information Studies Group 

University of Southern California Computing Center 
Los Angeles, California 
May 23, 1973 



291 



A. Wc McEachem 

Youth Study Center 

University of Southern California 

Los Angeles, California 

May 23, 1973 

Joe Fink 

Center for Urban Affairs 
University of Southern California 
Los Angeles, California 
May 23, 1973 

Robert Newman 

Youth Studies Center 

University of Southern California 

Los Angeles, California 

May 24, 1973 

George Mandanis 
Systems Applications, Inc. 
Beverly Hills, California 
May 24, 1973 

Paul R. Christensen 

Research Applications to Education 

Santa Barbara, California 

May 25, 1973 



APPENDIX F 



Demographic Items 
Principles of Supervision 

Age: 

Sex: 

Number of years of supervisory experience: 
Number of people who work for you: ^ 

Population of community for which you work: 

Type of work you supervise: 

How many supervision courses have you had prior to this course? 

In a typical day, what percent of the day do you spend on the following: 

Planning/scheduling work 

Coordinating/organizing work 

Directing/controlling work 

Training 

Monitoring/checking/inspecting work 

Dealing with technical problems 

Dealing with personnel problems 

Meetings with your supervisor 

Meetings with yi)ur subordinates 

Writing reports 

Writing work orders 
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Principles of Supervision 

A supervisor should (check the most important ^atements): 

spend most of his time inspecting work done. 

make sure employees understand company rules and policies. 

make company policy, when necessary. 

correct employees behavior in the presence of others if necessary. 

try to get other employees to handle his paper work. 

sacrifice quality work to get a greater quantity of work done. 

help employees to perform work when necessary. 

pass most problems "upstairs'' for decisions. 

let someone else worry about how well trained employees are. 

stay out of the personal affairs of his employees. 

let someone else worry about the safe or unsafe working conditions. 

pick only the most skillful employees for overtime work. 

minimize the importance of gripes. 

interpret company policy to employees. 

encourage criticism of company policy, where appropriate. 
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Job Functions 

The primary duties of a supervisor are (check the most important): 

checking work flow. 

inspecting quality of work. 

training employees. 

motivating employees. 

planning employee's work. 

coordinating employee's work. 

writing reports. 

doing employee's work. 

explaining his duties to employees. 

explaining his authority to employees. 

handling employee gripes. 

making procedural decisions. 

making company policy. 

committing funds. 

handling tools/equipment. 

setting company priorities. 

cooperating with superiors. 

inspecting working conditions. 

cooperating with employees. 

reducing waste. 

cooperating with the public. 

setting standards. 

giving complete instructions. 

evaluating company regulations. 
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Situation Number One 



Joe is a drill-press operator in the shop. His work is simple, and 
he is paid on an hourly basis. He has always kept his output up and has 
never caused any trouble. For the past two weeks you notice that his 
production has been getting worse each day, while the production of the 
men around him remained steady. As his supervisor, you 

(Put a plus (4-) by the two things you would be sure to do and put a 
minus (-) by the two things you would be sure not to do.) 

1. tell Joe to find another job. 

2. talk to Joe*s co-workers to determine why his production is down. 

3. warn Joe that if his production doesn't improve you will fire him. 

4. check with Joe*s family to determine if everything is Okay at home. 

5. talk with Joe about his performance. 

6. talk with your supervisor ab<^ut your problem witn Joe performance. 

7. retrain Joe for ariv^ther job. 

8. invice Joe out for a beer to discuss how things are going in the shop. 

9. encourage Joe to tell you of problems as soon as they -'irise. 




Situation Number Two 



As foreman, you are faced with a group of 14 employees who 
indicate they have some grievances; however, you refuse to call their 
shop steward and to talk to them as a group. Instead, you suggest 
that they appoint a spokesman. Before you leave, you threaten 
disciplinary action unless they are back at work in two minutes. 
When you return and find them still gathered, you 

(Put a plus (4-) by the two things you would be sure to do and put a 
minus (-) by the two things you would be sure not to do.) 

1' tell all the men to clock out and then you vill discuss their 
grievances with them. 

2. tell all the men to clock out and go home; they are suspended for 
two days. 

3. talk to the men about their grievances. 

4. set up a time to meet with the men and then tell them to get back 
to work. 

5. tell the men to go back to work; that you will talk with each man 
separately. 

6. admit that the disciplinary threat was a mistake and promise to 
listen to their grievances first thing tomorrow morning. 

7. pick the apparent leader of the group and chew him out right 
there* 

8. pick the apparent leader of the group, fire him, and then 
threaten to fire anyone who is not back to work within two 
minutes • 

9. tell the men to forget their grievances because management won't 
listen. 

10. ask the men to choose a grievance committee of three to tell you 
their problem immediately and ask the other men to go back to 
work. 
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Situation Number Three 



An employee, doubtful about following your instructions concerning 
a new method of loading a ciane lift, consults his shop steward, who 
informs him that he does not have to do the job if he feels it is 
unsafe. As foreman, you 

(Put a plus (+) by the two things you would be sure to do and put a 
minus (-) by the two things you would be sure not to do.) 

1. go to the shop and chew out the employee for insubordination. 

2. talk to the employee to try to find out why he thinks the new 
method is unsafe. 

3. get someone else to load the crane lift. 

4. chew out the shop steward for countermanding your instructions. 

5. contact an industrial research group to determine the safety of the 
new method. 

6. talk to your boss about the problem to find out what he would do. 

7. try the new method yourself to determine how safe it is. 

8. talk to both the employee and shop steward together to determine 
iXactly what the problem is. 

9. threaten the employee with disciplinary action if he doesn't do the 
job. 

10. encourage the employee to communicate directly with you rather than 
indirectly through the shop steward. 




Situation Number Four 



Your superior calls for you to report to his office. As you walk 
in, it is obvious that he is busy- He hands you a new order, and 
says, ''Take as many men off what they are doing and get this job out-" 
The phone rings, he answers the phone anu says, *'Yes sir," gets up 
and walks out leaving you standing there- This is about the fifth time 
recently that he has failed to give you complete instructions- You 
decide that you should 

(Put a plus (+) by the two things you would be sure to do and put a 
minus (-) by the two things you would be sure not to do-) 

1- get to work, obeying his orders to the letter- 

2- sit down and wait until he comes back so that you can get further 
instructions - 

3- call in one of your leadmen and give him the same instructions. 

4- go back to your office and hold the work order until you get the 
whole story- 

5- write a memo to him which indicates how difficult it is to do a 
proper job without complete instructions. 

6- begin getting the work done and then try to get the complete in- 
structions - 

7- call in a leadman and have him wait to get complete instructions 
from your superior. 

8. send a memo to your superior's boss telling him about the poor 
instructions. 

9. call in your leadmen and complain to them about how bad the 
situation is- 

10- go after him, and demand that he fill you in on the details right 
then and there. 
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Situation Number Five 



As supervisor, you have just spent considerable time planning 
the priorities of street re.palv work for your city. You are called 
into your superior's office at which time you are told of many com- 
plaints about unrepaired streets called in by citizens. You notice 
that these complaints are registered in areas you gave the lowest 
priority for work, which was based on analjizing the need, amount of 
work required, and uhe location. You decide that you should 

(Put a plus (+) by the two things you would be sure to do and put a 
minus (-) by the two things you would be sure not to do.) 

1. forget your plans and begin work on the areas complained about. 

2. show your supervisor your plans and make a strong argument for 
following them. 

3. ignore the complaints and follow your original plans. 

4^ ask your superior to help you replan the street repair priorities. 

5. ask your superior to provide policies and guidelines covering 
street repair priorities. 

6. do nothing until you receive specific orders regarding what areas 
should have priority for street repairs. 

7. criticize your superior for not giving you the information sooner 
that you could use the information in your planning. 

8. ask your supervisor to make a decision about what streets should b 
repaired . 
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Situation Number Six 



Your superior calls you into his office. He tells you that it 
appears that your men have not been performing their work up to 
standard in as much as he has received some complaints from customers. 
As a supervisor, you have carried out routine inspections of the work 
and found the quality to be acceptable. At this point you 

(Put a plus (+) by the two things you would be sure to do and put f» 
minus (-) by the two things you would be sure not to do.) 

1. tell your boss that he does not have the facts. 

2. try to determine what the problem is; what is specifically complained 
about. 

3. ask your superior to come and inspect the work with you. 

4. feel bad because of the reprimand suggesting that you haven't been 
doing a satisfactory job. 

5. decide that you will have to spend considerably more time inspecting 
work. 

6. decide that you will have to retrain the men to do better quality 
work. 

7. go and chew out your men for their poor work. 

8. decide to set higher quality standards for work. 

9. ask to speak personally to the customers with complaints. 

10. question other employees regarding the work of those considered 
unsatisfactory. 
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APPENDIX H 
Background Information 
Principles of Supervision 
(Do not put your name on these sheeics) 

Age J 

Sex: 

How many years have you been a supervisor: 

How many people work for you: 

What kind of work do you supervise: 

How many supervision courses have you had before this course 
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Principles of Supervision 

A supervisor should (check the most important sentences) : 

spend most of his time checking work done by others. 

make sure employees understand work rules and policies. 

correct an employee in front of others if needed. 

try to get other employees to do his paper work. 

give up the quality of work to get a greater amount of work done. 

help employees to do their work when needed. 

pass most work problems "upstairs" for action. 

let someone else worry about how well hi- : -^loyees are trained. 

stay out of the personal affairs of his employees. 

let someone else worry about the safety of working conditions. 

pick only the best employees for overtime work. 

pay no attention to gripes of employees. 

explain city policy to employees. 

cause employees to gripe about city policy. 
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Job Functions 

In the list below, check those jobs a supervisor should do: 

make sure work is done on time. 

check finished work of employees often. 

train clerk to use new office machines. 

encourage secretaries to try to make fewer mistakes in 
typing letters. 

arrange for machine repair ahead of time to lower the amount 

of worker down time, 

arrange work to make sure the paper work gets done on time. 

type finished copy of safety report to give to management. 

explain the job of the supervisor in passing on gripes to 

management. 

keep quiet about who arranges work and work plans. 

pay no attention to employee gripes about noisy working places. 

decide the steps to be followed in recording how money is spent. 

decide that secretaries in his department can leave work 30 

minutes early to miss rush hour traffic. 

decide to buy a new kind of electric machine without checking 

with management. 

unload trucks to help out workers. 

decide to cut down trees along a street without asking 

management. 
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Job Functions (continued) 

« 

start new coffee break policy even though the management 
does not agree with it. 

carry out regular check to find possible safety problems. 

listen to and put to work secretaries' ideas to make work 
better. 

answer questions that the public asks about changes in 
tax rates in a nice and careful way. 

set work standards for his department. 

check with new employees to make sure they have enough 
information to fill out personal leave requests. 

gather facts for management about possible cost-savings 
by using new machines. 
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Situation Number One 

Joe is a machine operator in a shop. His work is simple, 
and he is paid by the hour. He has always done his work well and 
has never caused any trouble. For the past two weeks you notice 
that his work output has been getting worse each day, while the 
work output of the men around him did not change. As his 
supervisor, you 

(Put a plus (+) by the two things you would be sure to do and put a 
minus (-) by the two things you would be sure not to do.) 

1. tell Joe to find another job. 

2. talk to Joa's co-workers to determine why his work output is 

down. 

3. warn Joe that if his work output doesn't improve you will fire 
him. 

4. check with Joe*s family to find out if everything is o.k. at home. 

5. talk with Joe about his work output. 

6. talk with management about your problem with Joe's work output. 

7. put Joe on another job. 

8. invite Joe out for a beer to talk with him about how things are 
going in the shop. 

9. encourage Joe to tell you of his problems as soon as they arise. 
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Situation Number Two 

As foreman, you are faced with a group of 14 employees who 
indicate they have some grievances, however, you refuse to talk to 
them as a group. Instead, you suggest that they appoint a spokesman. 
Before you leave, you threaten disciplinary action unless thay are 
back at work in two minutes. When you return and find them still 
gathered, you 

(Put a plus (+) by the two things you would be sure to do -ik-d nun a 
minus (-) by the two things you would be sure not to do,) 

1. tell all the men to clock out and tnen you x^ii.l discuss ^heir 
grievances with them. 

2. tell all Che men to clock out and go home; they are suspended 
for two days. 

3. taxic to the men about their grievances. 

4. set up a time to meet with the men and then tell then to get 
back to work. 

5. tell the men to go back to work; that you will talk with each 
man separately. 

6. admit that the disciplinary threat was a mistake and promise 
to listen to their grievances first thing tomorrow morning. 

7. pick on the leader of the group and chew him out right there. 

8. pick out the leader of the group, fire him, and then threaten 
to fire anyone who is not back to work within two minutes. 

9. tell the men to forget their grievances because management 
won't listen. 

10. ask the men to choose a grievance committee of three to tell 
you their problem and ask the other men to go back to work. 
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Situation Number Three 

An employee, afraid to follow your instructions about a 
new method of loading a fork lift, talks to another supervisor who 
informs him that he does not have to do the job if he feels it is 
unsafe. As supervisor you 

(Put a plus (+) by the two things you would be sure to do and put a 
minus (•) by the two things you would be sure not to do.) 

1. go to the shop and chew out the employee for not following 
your orders. 

2. talk to the employee to try to find out why he thinks the new 
method is unsafe. 

3. get someone else to load the fork lift. 

4. chew out the other supervisor for undercutting your job. 

5. get an outside expert to say hov. safe the new method is. 

6. talk to your boss about the problem to find out what he would do. 

7. try the new method yourself to find out how safe it is. 

8. talk to both the employee and the other supervisor together to 
find out exactly what the problem is. 

9. threaten the employee with disciplinary action if he doesn't do 
the job. 

10. encourage the employee to talk directly with you rather than take 
his problem to some other supervisor. 
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Situation Number Four 

Your boss calls for you to report to his office. As you 
walk in, you can see that he is very busy. He hands you an 
unfinished work order, and says, "Take as many men as you need off 
what they are doing and get this job out." The phone rings, he 
answers the phone and says, "Yes sir", gets up and walks out 
leaving you standing there. This is about the fifth time in the 
past few weeks that he has failed to give you complete instructions. 
You decide that you should 

(Put a plus (+) by the two things you would be sure to do and put a 
minus (-) by the two things you would be sure not to do.) 

1. get to work, obeying his orders to the letter. 

2. sit down and wait until he comes back so that you can get 
further instructions. 

3. call in one of your workers and give him the same instructions. 

4. go back to your office and hold the work order until you get 
the whole story. 

5. write a memo to your boss that tells him how difficult it is to 
do a good job without complete instructions. 

6. start the work and then try to get more complete instructions. 

7. call in a worker and have him wait to get complete instructions 
from your boss. 

8. send a memo to your boss's superior telling him about the poor 
instructions you get. 

9. call in a worker and complain to him about how bad your boss is. 

10. follow your boss and demand that he give you all the details 
right then and there. 
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Situation Number Five 

As supervisor, you have just spent a lot of time planning 
the priorities of street repair work for your city. You are 
called into your boss's office at which time you are told of many 
complaints about unrepaired streets called in by citizens. You 
notice that these complaints come from places in the city that you 
gave the lowest priority for work. You decide that you should 

(Put a plus (+) by the two things you would be sure to do and put a 
minus (-) by the two things you would be sure not to do.) 

1. forget your plans and begin work on the streets complained about. 

2. show your boss your earlier plans and make a strong case for 
following them. 

3. ignore the complaints and follow your earlier plans. 

4. ask you boss to help you replan the street repair priorities. 

5* ask your boss to give you guidelines to set priorities for 
street repair. 

6. do nothing until you receive direct orders about what streets 
should have priority for repairs. 

7. gripe at your boss for not giving you the complaints sooner so 
that you would use them in your planning. 

8. ask your boss to decide what streets should be repaired. 



311 



Page 10 



Situation Number Six 



Your boss calls you into his office. He tells you that 
it looks like your men have not been performing their work up to 
standard, because he has received many complaints from citizens. 
As a supervisor, you have carried out weekly inspections of the 
work and found the quality to be good. At this point you 

(Put a plus (+) by the two things you would be sure to do and put a 
minus (-) by the two things you would be sure not to do. 

1. tell your boss that he does not have the facts. 

2. try to find out what the problem is. 

3. ask your boss to come with you and inspect the work. 

4. feel bad because your boss says that you haven't been doing 
a good job. 

5. decide that you will have to spend much more time inspecting 
work. 

6. decide that you will have to train the men to do better work. 

7. chew out your men for their poor work. 

8. decide to set higher standards for quality of work. 

9. ask to speak personally to citizens when they call in complaints 
10. question employees about who is not doing good work. 




APPENDIX I 

The Service Bureau of Dearborn , Michigan 

The following information describes the service bureau which Mayor 
Orville Hubbard of Dearborn, Michigan, has instituted to handle citizen 
complaints. The system is presented because it offers a rather unique way 
to collect data on citizen complaints and to provide for follow through on 
the delivery of services to the citizen. 

There is no question that the delivery of public and social services 
to citizens of the community is a prime function of government. The fact 
that services are not being rendered as effectively as they should be has 
given rise to the development of community development agencies at both the 
local, state and federal levels. If, as we have suggested elsewhere (Com- 
munity Impact Evaluation) , a system of citizen complaints and the intervening 
response time to achieve resolution of those complaints, can be accepted as an 
indirect measure of the quality of life in a community, then the Dearborn 
system should be examined carefully. 

The maintenance of a quality service bureau in a community requires both 
a high degree of coordination among city departments, and a considerable 
amount of follow through. There is further little question that sustained and 
inspirational leadership to insure a delivery of services to citizens is a 
prime factor in the effective functioning of such a system. In the case of 
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Dearborn, Michigan, the mayor who has been in office for thirty-two 
years, provides the leadership and inspiration and the service bureau 
provide the necessary follow through on complaints received by citizens. 
The effective delivery of those services is achieved by means of a high 
degree of coordination among departments and by strong, effective leader- 
ship in the top executive position. 

In addition to the forms and the internal organization of city 
departments in Dearborn, that will be presenter in the material to follow, 
another important ingredient exists. The mayor has divided the city into 
districts. He has made each of his department heads responsible for a 
district. Each department head is required to tour his district at least 
once a week and file a monthly report with the mayor on the quality of life 
within his district. In this way the mayor receives direct information on 
the quality of life, the mobility of the population, the effective delivery 
of services, and is alerted early to potential problems. This surveillance 
and reporting technique provides additional information above and beyond the 
requests received and dispatched by the service bureau. 

The Dearborn system provides a model which offers means of collecting 
data on community indicators. The following materials have been provided by 
the city of Dearborn. Acutal copies of complaints received and their dis- 
position are presented as exhibits. 

(Information is to be typed here directly form the Dearborn material) 
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January 19, 1971 

SERVICE BUREAU PROCEDURE 

Almost all types of phone calls and requests are handled by this 
department. Thus, the poeple make only one call, saving the time of the 
individual being shifted back and forth. We take the call and give it to 
the proper department, division, or utility. 

We write an order for each request or complaint, consisting of 4 
copies. The Office Copy (white) is kept in an "Open File" until the yellow 
one is returned with a written disposition. Then the white copy is filled 
out with the disposition that was written on the yellow copy, stamped 
COMPLETED, and filed according to street address. The piJk or Division 
Copy is kept by the department who did the work. When the work is completed, 
the department sends back the yellow copy with a written disposition on it. 
The Director of the Seirvice Bureau reads each yellow copy to see that each 
has been completed, initials it, and then the white copy is pulled from the 
"Open File" as stated above. The Service Bureau's or yellow copy is used as 
the backbone for making the business reply cards. The business reply card 
is sent to each caller in order to determine whether the work was done satis- 
factorily or not. After the business reply cards have been sent, a personal 
permanent index card is made on each caller and filed according to their last 
name in our permanent file. This way we have a record of each call that comes 
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into this department. The Director's Copy (gold) is filed — after the 
Director reads it to determine whether prompt action is required — according 
to the department and the category of work being done. 

The following types of calls would be sent to the following departments: 



VECTOR CONTROL 



DOG POUND 



HEALTH 



PARKS & BLVDS. 



HIGHWAYS 



receives calls on any animal situation outside or 
inside the home in the case of squirrels, bats, rats, 
mice, bees, wasps, hornets, ants, bugs of any sort. 
Also handles problems concerning pigeons and poison 
ivy. 

takes all dog and cat complaints, and picks up dead 
animal of any kind. 

takes calls on unsanitary conditions INSIDE the home 
or building, and unhealthful food conditions in 
restaurants, stores, or business places. 

takes care of all city trees; trimming, removing, 
repairing, spraying, and inspection of same. Also 
gives advice to residents on private trees. Tree 
stumps are removed within a month after the city 
tree has been removed. They also pick up branches, 
shrubs, and even entire trees that have been cut by 
the residents themselves — although there may be a 
charge of which the caller is Informed. If the work 
was done by a contractor, the contractor must take 
it away. 

Also takes care of city parks, weed cutting in vacant 
lots, and lawns damaged by snow plows. In the winter, 
it handles the plowing of snow from sidewalks. Parks 
& Boulevards is divided into two sections — East & West- 
each end of toxm taking care of its own calls, 

picks up broken concrete, rocks, bricks, lumber, dirt, 
clay sod, clothespoles, garage doors, storm windows 
& screens, and any other article which cannot be 
picked up by the regular Sanitation truck except 
for large appliances and furniture. It also takes care 
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of street maintenance (cleaning, sweeping, patching), 
slagging of alleys, cleaning & checking of sewers and 
catch basins; and, in the winter months it handles 
the plowing of snow from the streets and paved alleys. 
Calls concerning flooded basements are sent to this 
department so that city lines can be checked and advice 
can be given to the resident. 

Charges are sometimes given on excessive amounts of 
debris and the caller is notified of this. 

picks up all garbage and rubbish weekly — each area 
having a particular uay. Grass clippings, leaves, 
should be in containers and picked up on regular pick- 
up days. A special pickup is required for carpeting 
(unless it is cut into 4 foot long rolls and tied), 
furniture, and heavy appliances which is then forwarded 
to Mr* Burich for pricing. Sanitation also handles 
complaints for burning rubbish, or untidy alleys, 
damaged trash cans, and complaints regarding pickup 
men. 



takes inspection of run-down buildings — houses, garages, 
etc. They check violations of residential zoning; 
repairing cars on street, driveways, or in garages. 
Also checks on ill-kept yards, vacant homes, disputes 
over fences or eyesores. One of their check-chief 
duties is inspection of all buildings during construc- 
tion for their permits, etc. Complaints regarding odors 
or smoke in the air is handled by them. Especially 
those concerning incinerators. Heating problems go to 
Mr. Horschak. 

takes care of all sidewalks and curbs done by the City 
or by Contractors — handles all construction work done 
in the City. 

handles calls pertaining to the plowing of lots (pri- 
vately owned) and the planting of new trees on city 
property. All these cal.ls are transferred to DPW. 

handles stop signs, yield signs, etc. They are approved 
by Lt. Llndsey at the Police station and then sent to 
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TRAFFIC SAFETY : 
PARKING METERS : 

POLICE : 

PUBLIC UTILITIES 
& TRANSPORTATION 

MAINTENANCE : 

LEGAL or 
CORPORATION 

COUNSEL : 

SIGNAL BUREAU : 

WEIGHTS & 

MEASURES : 

MAYOR'S OFFICE : 



Parking Meters for installation. 

handles complaints on high hedges obstructing view 
of drivers, and larger signs for traffic safety. 

handles repair or replacement of street signs, church 
signs and direction signs, as well as street cleaning 
signs. Officers also inform store owners to clean 
litter from around their establishment. 

handles requests for the installation of signs, com- 
plaints regarding cars parked on the street, shopping 
cart complaints, excessive noise, large trucks blocking 
traffic, abandoned autos, as well as complaints where 
immediate action is required. 

handles the repairing of street lights and telephone & 
•.las company problems by notifying companies responsible 
Also takes care of complaints on bus shelters and issue 
bus passes for Senior Citizens. 

handles the repairing of lawns after they have been 
damaged by city snow plows and repairs city fences, 
as well as other repairs needed on city property or in 
city buildings. 



handles all legal problems and consults with citizens 
on legal matters pertaining to written complaints & 
suits. Also handles complaints on residents receiving 
literature from businesses when their homes are posted 
with "No Handbills or No Soliciting" signs. 

handles complaints on operators and street traffic 
lights that are not working properly. 



handles complaints of large trucks going down resi- 
dential streets posted with "No Commercial Vehicles" 
sigi;s. Also handles inspection of scales in business 
establishments . 

handles all complaints on Camp Dearborn and many sug- 
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gestions made by residents, 

FIRE DEPARTMENT: handles complaints having anything to do with the 
health, welfare, or safety of individuals within 
the City of Dearborn, but these are usually referred 
to Fire Dept. from Building & Safety. Also takes 
care of fire hydrants, 

handles complaints on sub-standard homes in the south 
end of Dearborn and also handles Eugene and Porath 
homes. Also takes care of any complaints made on the 
house demolition crew working in the South end of 
Dearborn. 

CITY PLAN : handles complaints on demolition crews working on 

homes in the east and west end of Dearborn, Also 
handles the closing of alleys. 

At the end of each day, a 3 x 5 card is typed totaling the number of 
written requests and are listed by department in this order: Sanitation, 
West Parks, East Parks, Highways, Building & Safety, Engineer g, and Miscel- 
laneous (which includes any requests sent to departments other than those list- 
ed in the above) with the total of requests made to each department listed. 

About the 15th of each month, a report is made for each department listed 
if there are any requests that are past-due (those which have not been com- 
pleted within a month). This is typed in duplicate and signed by the director 
and then one copy is forwarded to the department involved \/hile one copy is 
kept in this office. The departments acknowledge these and forward them back 
with notifications as to whether the work has been done or not with the date 
of completion. 



COMMUNITY 

DEVELOPMENT : 
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To Moyor Hubbordt , .13-73 #77217 vg 

The recent attention I received from the Service D'-portment wos: 
Re: Replaced City tree h-l6-73/anita/rice-vigilante/e» parks 
H Sotisfoctory □ Unsotisfoctory Suggestions 

^7^^^.<. ^^ .-/l-O ^^c...^ -/^-vV^' -r^ /-i^ ..^^ . 



Address_i:^il^:i^ Phone X^^A 

Mra. Staiacy Zalewski ^ ^ 752i| Appoline 26 - UJ 1-396U 



7AT/7.'3h:i, Stanley J. ('73-67) M^ecLn \. . 

7^;2!! .\rrolj.nr5 2*^ L^^ l-396!i "vSAI-PK P77i30::AL C VPuD - 

for 3cnior Citizen" 
7_.03.6P /'COI67 Pick VP -\;?noa ri^^sb 
7-.23-^P Picked stray ham; ster/dawdy/al 

li«27-''^r j(->^jlir? Trin trce/lh 
2-2)j-69 Trimr^ed troe/rice-asj-lunch/ al 

^wll--''')9 ;>*]3h2n Chock C'fj..r\-\ trec/cd 

6^12-69 ' Trne shovrinr^ sirns of D.E.D. Sxr^lained 

that if tree r:ets vorse, v;e vill remove/ 
broda/al 

7-29-^9 il\^-X^^:> tei-iove diGoa2'':d trt^yal 

9- 5-69^ Reinoved tree stimp/rice-cj n?qrelli/al 

10- 12-72 ;y)r)i21 Check new tree/al 

10-12-72 Chocked ad -ised, minor danare. She 

vjll nail back jn Srrj np/ri.ce/al 
(over) 
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DATE ''-^-3-73 

time1 0-3' : 



It Takes Teamwork To Get Things Done 

REQUEST FOR SERVICE 

SERVICE BUREAU 



WHITE . OFFJCE COPY ^ ' 

YELLOW. SERVICE BUREAU'S COPY 
PINK . DIVISION'S COPY 
GOLD- DIRECTOR'S COPY 



N2 77117 



PARTY CALLING : -^T: . ::tnnlo:/ ^ Zalev:ski address 752)4 ■"T>polina 



26 

ZIP 



PHONE 



NATURE OF REQUEST Li"t voar the cj.ty removed a treo nexrt door to her and in tho rrocoss^ 
we dannnod hor youn^:: city tree. The foreman who clieckcd her tree last year told her that if 
she wasn't satlg.':^ed with it^ V7e would replace the tree^ Please check young tree and sen i f 
it needs rcr] ?c:.:->-^ r.rJ rf '^rill replace it. taken ry anita 



DISPOSITION Replaced tree. 



CALLED IN TO 

REFERRED TO 

Sanitation 

West Parks O 

West Highways 



East Parks 
East Highways 

Bldg. dk Safety □ 
□ 



DATE PARTY NOTIFIED OF DISPOSITION. 
SIGNED Rico/Vigilante 



PARTY'S COMMENT: Satisfactory Q Unsatisfactory q 

. DEPARTMENT P^^^S DATE 5-16-73 



PARTY CALLING 



USE REAR OF SHEET FOR FURTHER COMMENT. 

KM' 

ADDRESS 



ZIP 



PHONE 



NATURE OF REQUEST . 



TAKEN BY_ 




CALLED IN TO — 
REFERRED TO 

Sanitation 

West Parks Q 

West Highways □ 



East Parks 

East Highways □ 

□ 



Bldg. (Ik Safety 



□ 



DATE PART)flNOTIFIE|/OF DISPOSITION 



SIGNED 



PARTY'S COMMENTj/^ S<Ujrfactory Q Unsatisfact^y 

DEPARTMENT (/^ / "P^ "^^T", 



USE REAR OF SHEET FOR FURTHER COMMENT. 



ERIC 
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Stan Hughes 
12 April 1973 



MONTHLY DISTRICT REPORT 



COKE SIGNS 
HOUSES FOR SALE 



: Area bounded on the North by the 
South side of Ford Road, on the 
East by the West side of Wyoming, 
on the South by the North side of 
Michigan, and on the West by both 
sides of Orchard 

: Total — 9 

: Total — 17 (total last month 7) 
5537-5539 Calhoun (Owner) * 
5238 Reuter (Matuszewski) 
5452-5454 Calhoun (Sassanelli) 
5100 Neckel (Owner) * 
5458 Maple (Owner) 
5238 Maple (Christie) 

5034 Maple (integrity) * 

4883 Maple (Fordson) 

5218 Horger (Dearborn) 

5264 Middlesex (Forbush) 

5473 Williamson (Vincent Nee) 

5489 Williamson (Garling New home) 

4811 Argyle (Earl Kein) 

5018 Kenilworth (Garling) 



5131 Kenilworth (Real Estate One) 
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4760 Roemer (Leslie) 
4746 Chase (.Owner) * 



DAMAGED SIGNS 



: Total— 6 



Sent #77111 to P.M. on 
4^l2-73/al lj2jS^&4 
REFERRED TO COIT.TY b&6 
REPAIRED/4^l3-7i^/ 
loalevyoh/dm 



PRIVATE PARKING LOTS THAT 
NEED CLEANING 

Sent #77112 to B&S on 
4'^12'73/al 



Ford Road, Eastbounc at Wyoming 
"This Lane Must Turn Right" sign 
down (State Highway Sign) 

Wyoming, Southbound at Michigan 
damaged "US 12" sign (State 
Highway Sign) 

5408 Reuter, sign pole bent 
4798 Williamson, sign pole bent 

Total~2 

Club Chablis, 12900 Michigan, 
rear lot 

Diamond Jim's, Northeast corner of 
Michigan and Hartwell 



HOLES IN RO/D : Total~l 

Sent #77113 to E. Highways Schaefer, Southbound, Colson to Michigan 

on 4-12-7 3/al 



JUNK CARS IN YARD : Total— 2 

Sent #77114 to Auto Squad on 5402 Argyle 

4-12-7 3/al OFFICERS TALKED TO 

JOYCE HUNTj 582-3329^ 5416 ARGYLEj WHO STATED THAT THE VEHICLES WOULD 

BE REMOVED AS SOON AS POSSIBLE. CAR- 3 4-15-73 HARWORTH & MELTZER/4-'l6-'73/al 
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NO BUILDING PERMITS DISPLAYED : Total— 2 

Sent If??ll5 to B&S on 4-12-7Z/ 5101 Kendel 
al VIOLATION ISSUED TO SlOl 

KENDAL. PERMIT ISSUED FOR 5121 Jonathon 

5121 JON THON—NO WORK STARTED, 
/hemp 4-lZ-7Z/al 



STATE LAND THAT NEEDS TO BE 
CLEANED OF LITTER 

Sent #77116 to City Beautiful 
4'l2-7Z/al 



Expressway area, Michigan Avenue 
from Wyoming to Miller 



324 



